This is ignoring that Barristan has a sort of arrogant pride of his own. Being rejected by the most powerful ruler in the land after serving kings is something he's never actually had to cope with. From the age of ten, he was told that he was a standout example of his respective archetype, a notion continually reinforced by his own skills and accomplishments.
That Viserys finds no use for him, not even as someone who bears a sword and fights for him--and if we did, we would have offered him a Legion cloak--is particularly galling. It would have been an insulting offer, since a Knight like Barristan would think of paid professional soldiers as akin to their closest cultural contemporary--mercenaries (honorless, cut throats, turncloaks).
Rather than having more honor than him, since they all swear oaths to fight our enemies, which we openly admit include fiends and monsters. That's not a career you get into without being fairly loyal to your nation and leaders.
At the end of the day, the grand irony of Barristan Selmy is that he derives his own value through the lens of being regarded as useful to the person in charge of everyone else, no matter their morals or personal inclinations, and then masks all the wrongs he allows to happen in front of him and maintains his reputation by saying "because I am following my oaths, I'm really a good man inside".
Barristan justifies
later on to himself and to Dany that he didn't choose to serve Viserys because (in hindsight apparently) he was too much like Aerys.
In this quest,
@DragonParadox highlights that this was Viserys having a childlike admiration for his father, being a quite sheltered child without much information about what other perspectives are like. He was told his father was a good man, and believed it, yet after having to branch out and take the world on by himself, he definitely grew beyond telling himself pleasant lies and trying to convince people it was the truth, and instead much preferred to be honest even when it wasn't in his best interest.
Barristan couldn't have done much in this situation, even with the best approach of saying he served who he viewed as the likely winner, and did what he reasonably could to be loyal to whoever he served. It would have gotten him a big fat nothing sandwich, but we'd be a hypocrite to make a big deal out of such honesty.
But that would clash with his entire identity, revolving around him being an exemplar. Again, he was escorted by his own prison guards to meet with Viserys, showing that absolutely no one in his life has actually realized who he is, because he's got this delusional ability to project this false narrative overlaying what he's done and what he'll do and fabricate this alternate reality where he is a righteous and virtuous soul, fighting for a good cause.