No, but joking aside, what he got were the holdings, not the people. So he owns all of the houses, all of the shops, all of the material goods, and so forth. He'd be entirely within his rights to toss everyone out. He probably won't do that, though. Basically, the old guy set up an enticing little lure to cause Xi Fu to adopt himself.
That said... honestly, I don't feel like Xi Fu wants to own a bunch of holdings in the city. He's going to want to make sure that the people he cares about are taken care of and that's basically it. If anything, I'd expect him to basically pass that stuff directly along to his boss. She'd wield it better. She has her own pet mercantile cultivator, after all, and is also better suited to leveraging the political influence.