@Snowfire,
@Azel Honestly, the biggest barrier here is, Viserys could
say: "Your son seems more than smart, he seems
erudite and I adore that. My City is a center of learning, and I myself am a bit of a scholar, and that besides so are many who serve me. He could do well here if you fostered him."
You know what Randyll could have done to Sam rather than say, something more sane (and
useful politically, if nothing else)? Send him to the Citadel.
You know what he
didn't do in canon, and likely still wouldn't?
Send him to the Citadel. He sent him to the Wall, because he holds such interests in contempt. If he has to get rid of a coward of an heir, neither could he do it by making him take up the Chain, because "no son of mine will be a robed bookworm. He's going to freeze his ass off in a place where they will force a sword into his hand, one way or another, and if he runs away from
that like he did in Horn Hill's practice yard, they'll fit him for a billet, a rope or a block."
Just say you'll teach him to be brave. It's the only thing compelling to Randy.