The Grey and Amethyst Colleges just look like abandoned buildings to uninvited guests. The Celestial and Light Colleges require a decent level of magical ability just to be able to see them. And the Amber College isn't in Altdorf, it's out in the Amber Hills. The only three accessible Colleges are the Jade, the Gold, and the Bright. The Jade College has forty-foot high walls, is full of plants willing and able to assist in its defence, and is fueled by the Altdorf Nexus. The Gold College is full of experts in creating things that explode and is nestled into an extremely defensible position flanked by the city walls and the river. The Bright College has walls and defensive enchantments but really the main thing about it is it's full of Bright Wizards.
Imagine trying to assault a hedge maze that hates you, or a crossfire of every substance on the continent that can be made to go boom, or the elemental plane of burning, while four whole Orders are able to sally forth at will from places you can't even see, and you're not allowed to damage the city because that's where Dieter is trying to get his hedonism on. Imagine doing so with zero material support from the local authorities because, again, Dieter. Imagine doing so knowing that the local Taalites and Rhyans like the Jades better than you, the local Verenans like the Celestials more than you, the local Shallyans like the Lights better than you, the nobles like the Golds and Celestials better than you, and the masses are much more scared of the Greys and Amethysts than they are of you.
The Wizards are backed into a corner, and it's a corner they've spent a century attuning to their Wind and filling with weapons and enchantments and secrets. They're fighting for survival. You're fighting for an Emperor that fled from Nuln and let it be destroyed by a Waaagh led by a Goblin. Your hands are tied in a dozen different ways, and the Wizards just had every restriction placed upon them repealed. There is no longer anything they can do that is more illegal than their existence already is. All those spells that come with dire warnings about frivolous use, all those summonings that resemble Daemonology to the untutored, all those rituals that call upon powers not invited to the Grand Conclave, all those artefacts too irreplaceable to risk under normal circumstances? They've just had all their reasons not to use them stripped away.
...oh dear, I've just received word there's a very dangerous witch that needs hunting in a place that is very not here. Well, I'll leave you chaps with it, I'm sure you have it under control, best of luck to you, let me know how it turns out...
