Because Witch Hunters reckoned that the Cult of Vylmar was just a gateway/front to a Slaaneshi cult. From what I recall from Tome of Salvation, Empire theologians also couldn't find where the Cult's strictures and writings had even sprung up from, so the Cult didn't really have much in the way of legitimacy beyond the patronage of nobles and wealthy merchants.
Needless to say, they were pretty quick to abandon ship when the Cult of Sigmar at large started glaring at them and reaching for the hammer.