I'm not sure it will. The worry here isn't that they would come in and take over our land, the worry is that they'll raid us for our knowledge. Divine Stewards shouldn't trigger because it's not like we're causing environmental damage; Lord's Loyalty shouldn't trigger because the miners are safe. Plus, if they actually stay there and take over the mine proper, they make themselves a juicy stationary target that we can then surround at leisure, instead of turtling up defensively. As I understand it, Lord's Loyalty should only trigger if we fail to do something about it after we know there's attackers at our doorstep. Tearing down the kilns and pulling back the miners should address the "raid us for knowledge" problem without requiring an immediate defensive commitment. Best-case, they run in, raid us, find nothing much of interest and, disgruntledly leave. Second-best case, they try to take over the mine and make themselves a stationary target for us to go grind up. Worst-case, they get angry and commit punitive atrocities against the civilian populace nearby. That seems relatively unlikely though.