It is the best job for research, imho. It has a big budget and it keeps the grey college off our backs. I want to lose neither of those things.
The Bursar auditing worry and the grey college attempting to assign us somewhere else worry are different worries and they find me on different sides of the aisle. I allready stated my thoughts about the Bursar worry: realpolitic makes attacking us unfeasible. But reassignement is a very different fruit imho in that it is entirely feasible and cost effective.
You see, the grey college has a LOT of open fronts, so if they spot a member being unproductive for the Empire, they slowly amp up the soft pressure until they start being productive. This is significantly safer than a kill order (nothing we can do if our favours start dissipating, or if we suddenly cannot purchase certain books, or if we lose battle magic privillege, or if we face several administrative obstructions). It is also significantly more likely to happen to the powerful, because they want their top agents to be productive, and they know that even the most competent grey mage wants to just stop doing things for the Empire and just research because that is what mages like to do.
They likely won't send us to wipe the ass of some incompetent noble, they'd pointedly pressure us to do something, and the more we do not, the more they will suggest certain things. As things stand, however, we benefit the Empire tremendously just by being a Loremaster on K8P, so they do not want to move us because they'll lose their dwarven relations generator. It'll be a different story, however, if we quit, because they need all hands on deck, and they will use soft means we cannot retaliate against without being the bad guy to do so.