He doesn't have to even notice us to instagib us. If Mathilde miscasts anywhere that would be considered on the same battlefield as him, she's instagibbed. He also, lorewise (the game is designed to be balanced), can't be injured permanently by normal means. In lore, killing him was beyond the Avatar of Isha, who had to draw on the forest's power to do so, and even that wasn't permanent. None of the elves weapons could even harm him.
Also, given that the winds look like a kaleidoscope, the chances of us miscasting seem even higher than normal. And note that we probably are usually okay with slight miscasts occasionally, but even a slight miscast is insta death.
Casting magic is stupid under such conditions. Don't do it until we are sure that it is safe. Fortunately, by waiting a day, we can see what happens to spellcasters from the hopefully attacking other tribes. If we don't see magic cast, that also tells us that they had good reason not to bring the shaman, or the shaman knew not to cast, so that's evidence it's not safe.