We just need more carrying capacity. We can come up to over 100 tons.
Granite has an average weight of 175 pounds per cubic foot. 100 tons of lifting capacity would allow us to move approximately 1143 granite blocks.
Let's double that to 2286 granite blocks with measurements of 12x12x6 inches.
A single 10ftx10ftx10ft room composed of such blocks would use 573 of those blocks, assuming one wall has a 3ft wide by 6ft high door and a 3ftx3ft opening in the ceiling for a spiral staircase.
Let's stack several of those rooms for a more substantial structure.
The 2nd floor would need 467 blocks, assuming each wall has a 2ftx3ft window opening and another staircase opening in the ceiling.
Another 467 blocks for the 3rd level, and then 476 blocks for the 4th and final level.
That's 1983 blocks worth of granite, for a total weight of 173,512.5 pounds. We now have 26,487.5 pounds left over for a sturdy iron door and spiral staircase, with iron-barred and shuttered windows, and whatever furnishings occupy the structure. Or maybe to put a nice peaked roof on it?
I know this is a simplistic example, and not totally representative of stone structures, but we ended up with a small 4 level stone tower for our 100 ton lifting capacity.
That is not a very large or substantial structure, and it would require a full array of buffs to have a chance of Teleporting it.