Most of them in that exact situation probably did die, yes, but it only takes one or two, and those who manage to survive are disproportionately likely to live longer lives via superior cultivation. Basically, do you think that the Southern Tribes are COMPLETELY ignorant about the empire, or that they are not completely ignorant about the empire. Because if they aren't completely ignorant about the empire, then like the first thing they will know is that if they ARE north in the territory that the empire operates in, then are not going to be treated any different than the northern tribes are when they DO run into the empire.