A couple of thoughts here:
Using only the anime as a guide, I think we can assume the Gate has opened up in our world at least twice before Ginza and perhaps several times. We know that we have an offshoot of the Roman Empire. This assumes that a viable population went in sometime before 380 CE If Hardy is responsible for opening the Gate into Roman territory, she obviously got more than she bargained for, showing that the deities in the Gate World certainly are far from all powerful. If Hardy had any comprehension of who she was letting in, she probably would have picked a different spot.
Why? Romans are not history's most feared warriors. But they are renowned for being organized. They had to set up a viable community and defend it and then expand out in a way that none of the other races could match. I think we can assume that other groups of humans came through prior but lacked the organizational and engineering skills that made Rome so formidable. The limiting factor for these transplanted Romans would have been in building up numbers. In our world, it took 700 years between the founding of Rome to the Proclamation of an empire. In the Gate world, assuming their years are roughly consistent with ours, the Empire was proclaimed about 1330 CE our time.
Some would say this suggests that the Roman arrival would be considerably later than 3rd or 4th century CE. Assuming that it took a similar amount of time between arrival to proclaiming the Empire as from founding Rome to the reign of Augustus, that would give 700 + 687 =1387 years, subtracted from our calendar, say 2016... 2016-1387 = 629 CE as the approximate year of arrival.
The only problem with this is that Romans of that time would have brought along their State Religion: Christianity. Instead, we know they brought the old pantheon with them. How do we know this? Her name is Rory Mercury. So the Roman part of the Empire's heritage had to have arrived before 380 CE.
But a Christian group DID arrive through the Gate eventually. This took place no later than 1450 CE. Gunpowder and printing existed in our world at that time, but not with the population that went through. These people had a major impact on the Empire and it seems that while their religion did not survive, they became a significant group in the Empire. These people were English and some of them nobility. We see it in the symbolism of the Rose Order. In the adoption of the English bow, and in the name of Pina's page, Hamilton. We also see it in how fast the language barrier is overcome.
In our world, Latin has morphed immensely into an entire family of languages. I expect some forms of Latin are in use in the empire but would Itami recognize them? One language used by the Romans that is still fairly recognizable today is Greek. Greek was the language of the learned and may also be found in the Empire, but again, why would Itami recognize and understand Greek? In episode 5, it is clearly shown that Itami understands what Princess Pina is saying except the odd word here or there. That makes the most likely candidate: ENGLISH.
Certainly not our modern English, but Middle English of say the 1400s would be recognizable. It isn't Shakespeare but the language of Chaucer would be something that someone who knew English could get a general gist of even if the written forms would be trickier Also both our English and theirs would have been drifting in the same general direction due to the influence of languages of the Romance family. Specifically French historically and the mutations of Latin that exist in the Empire world. So at some point, the Empire received an infusion of English blood at the top.
Just a few thoughts