That isn't the case; you're putting words in the mouth's of his supporters here, and in a bad way. The point being made is that, by the good we are doing through conquest, it makes attempting to dislodge us far worse for all involved. The destruction of our polity, and it would have to be nothing less than that, would send all of Essos into a tailspin. It would accelerate the growth of his enemies and actively sabotage the spread of benevolent order that Archons champion.
It isn't that anyone is rejecting that he's considered killing us - as far as I can tell. It's that killing us would do far more harm than good. So long as we continue to expand in the same way as we're doing so: breaking slavery, establishing education, and all the other marks of civilisation that we're bringing with us, then the balance of probability will remain in our favour.
The idea that he'd think he could just replace us and not have the entire Imperium fall apart is farcical. Yes, he's thought about it, but a major part of his calculation will be the good of the people we rule, instead of his own benefit. You seem to be confusing some of his motivations with those of Zherys, and that's really going to throw your math off.