Considering how much of the rest of the continent is blasted wasteland or impenetrable mountain or jungle, a large arable plain that's about a third to half the size of Rokugan inhabited by primitive screwheads is a pretty attractive target. It's even surrounded by distant mountains populated by MORE of them.
The Yobanjin are an issue yes, but if you set up a group with a mandate to conquer them over an extended period of time then it'll probably be fine. I say this because without the Kami their technology and magic falls drastically behind over the years. At the Dawn of the Empire, where we are now, they're somewhat of a threat, but they're also not in any way entrenched. Later on their weaponsmithing is vastly inferior, but they know the terrain fairly well. It evens out.
Basically, it's an area that could be exploited and has little in the way of natural defenses, rendering things down to a people vs people fight, and the Rokugani have a massive edge over the Yobanjin in that arena.
