You know, no one's brought up the nature of 40k so far. 40k's purposes as a setting are, in order of priority:
- A setting for a tabletop army game that doesn't want you to feel too bad about playing as or against any one faction, and thus makes certain all are uniquely awful.
- A setting for a tabletop rpg that presents a boatload of local and regional problems that player characters can conceivably fix during a campaign without unduly affecting the greater setting.
- A pulpy dark Sci-Fi setting for novels.
- A self consistent setting that lends itself to dissection and discussion.
I bring this up because each of these purposes lends a different outlook on what authorial intent is in regards to the Imperium's necessity.
Looking at 40k through the lense of purpose 1, yes, The Imperium is the only way humanity can resist chaos, because playing
against a xenocidal police state is the only way to make playing
as mindcontrolling daemons or world eating bugs feel guilt free. Everything needs to be awful, forever, and the Imperium fits that. Also, GW doesn't sell alternate human faction miniatures. That's 40k's authorial intent, at least as much as a tabletop setting with multiple author's and writers has one intent.
However, seen through the lense of the second purpose, a setting to set original rpg campaigns, civ quests, or fanfiction in, authorial intent doesn't mean jack shit. Sure, a Mary Sue human civilization is hiding behind a warpstorm and is ready to kick chaos ass, do whatever you want, 40k is a setting at least partially conceived as the base for fanfiction.
The biggest misunderstandings of 40k probably derive from the third and fourth purposes, since novels and wikiwalks are probably the only way most of us experience 40k. Unlike serious, standalone novel series the novelizations aren't the setting's Bible, they're fanfiction good enough to get the GW seal of approval, and serve to flesh out the world and populate it with characters for individual player's rpg campaigns/tabletop army battles/shameless ultramarine smut. You want a legendary, high level commissar with a full backstory and complex personality to show up and wow your PCs? We got a couple pre-made for yah! Want a terrifyingly powerful elf warlock? How asshole-ish do you want em?
All of this of course comes at the detriment of an internally consistent setting that always has the same themes and narratives, and can be easily discussed in a... *checks* ... Fiction discussion forum.
@BobTheNinja does that help?