The impression I always had was that the Skaven were more simply more concerned with fighting each other than anything else. They don't think the Empire is a threat because it isn't. They almost regard the dwarves as a threat, but mostly as 'annoyingly tough to crack'. They may fail to sack a Karak, but there's nothing the Dwarves can do to skavendom itself. The elves... they don't interact with the elves much, but they fall into a similar category. But another Skaven clan? That's an existential threat. That's something that can directly fuck up your clan and kill you, personally, if you piss them off enough.
Sure, they could chuck a doomrocket at every major Imperial city and send in fifty million clanrats to mop up what's left*, but that would be enough of an effort for any single clan would exhaust itself - and therefore be destroyed ... they'd have to cooperate in an environment where plenty of backstabbing is possible. Each clan would be so busy jockeying with its rivals and making sure no one is undercontributing or over-taking, and they'd pretty much immediately fall to infighting against the true threat, each other. This is... pretty much exactly what happened when they fought Mandred - they were flattening the Empire, it was just a question of how long it would take - but their commitment allowed one of the clans to stage a coup. And they were weaker then, coming off a long war with the Dwarfs and without the technical clans (well, Pestilens was the one that did the couping, but... you know what I mean).
No, much better to leave the man-things to their ignorance, raid them occasionally for food and treasure, but ultimately - they're not a threat, they know it, so it's better to focus on actual priorities, like that INFERIOR CLAN OVER THERE THAT'S DOING RATTING WRONG. Once all the lesser clans are dead and <INSERT WARLORD HERE> has been acknowledged as the greatest of all Skaven, then the destruction of the lesser races is a fait-accompli.
*End Times, yeah, but note this is exactly what they did in Vermintide.