See, the 'rest of the Kingdom' I can buy. But saying that our family would be eager to get rid of us, that's quite strange. Elza will never believe Anna would abandon her, and even if her parents are afraid of her, her sister would shield her. Even from a pragmatist's perspective, her father doing anything to Elza would mean losing both heirs, as Anna would never go along with it and won't forgive them for it. Seeing as they have no more children, that's arguably more detrimental to the Kingdom than whatever danger Elza might potentially present in some undefined future.
They took an utterly wrong approach to the situation, claiming that we need to be freed against our will to fulfill some kind of great destiny that we don't know and thus don't care about, and claiming we are unfit to decide for ourselves because our mind is trapped as well. They have done nothing to convince us otherwise, other than believing than what they themselves are saying, and considering wanton destruction (that might have harmed people, not necessarily Anna) to get their way which throws more doubt on their words. While Elza's power is dangerous and unpredictable, she herself is no murderer (yet?).
Were they to offer this as a way to master the power and make it safer, as a way to ensure those would not hurt Anna or anyone we would not want them to, then perhaps Elsa would seriously consider it. But as things stand, we have no reason to believe they even have those concerns. We have no idea what they will be trying to do with our powers, only how far they are willing to go.
As for how likely it would be for us to meaningfully resist them, that's a different matter entirely. I only gave my reasoning for what I think Elza would want/not want to do.