People often mischaracterize Roman landowners as parasites.
The actual situation of Roman agriculture varied drastically. When speaking about the early Imperial period; In Egypt and near-Asia the old Hellenistic system was maintained, in Italy and Greece there were lots of slave driven farms, Celtic areas maintained their old social order and in the border regions there were a lot of independent farmers.
Even "wealthy patricians owning everything leaving nothing for most others?" doesn't mean what you think it does. In most cases a patrician owned various pieces of land across the Empire or provinces, in order to isolate themselves from a bad harvest in a region, these areas were managed by a local representatives who hired workers, leased away land and managed the area. This allowed landowners to pursue intellectual or administrative careers. Of these landowners the Emperor himself was the biggest. The times when people were forced from their land were when a landowner decided it was cheaper to use slaves or wanted to grow luxury products that needed specialized workers to grow. These landowners of various sizes would then take the money they made from these crops and became the main financer of the craftsmen living in the city, who could be described (in most cases) as living a step above poverty.
80 to 90 percent of the population of the empire was at some point involved in the production and movement of agricultural products. The landowners left enough for other people to sustain themselves. Heck, being a farmer of any kind (baring a slave) was viewed as being better than being a minor trader or a regular craftsmen.
Serfdom itself stems from the reforms Diocletiánus (around 300 AD) made. After the period of the Barracks Emperors, he reformed and expanded the army. This required a new tax system, leading to a system based on the amount of land you had, the quality of the land and the amount of people that worked it. People in the cities had to pay taxes individually depending on their occupation, rather then it being based on land. In order to stop people from evading taxes by leaving land or switching occupation (and therefor evading taxes) the farmers were bound to the land, the city dwellers (hereditary) to their occupation and the ruling low level elite to their positions. For the record the up to this point the ruling elite also had to pay a deposit on the taxes they collected up front from their personal funds. Where in previous times these positions were a sign of prestige, they were now a crippling burden. This led to nobody wanting the positions and forced Diocletiánus to expand the the provincial and Imperial administration, drafting primarily from the army, militarizing the bureaucracy considerably. Aside from the army, Diocletiánus drew from the knights (Equites, Lower aristocrats) to man this expanded administration. Which in turn led to the major landowners, often senators, being excluded from both military positions and goverment positions (aside from the senate).
The way these taxes worked stimulated local governments and landowners to keep the best talented people for themselves while sending the "worst" of when the army came recruiting/drafting. This forced the army to rely more on cheap Germanic mercenaries. As a compromise the Emperor allowed the landowners to pay in money instead of manpower, since they could use the money to hire more mercenaries.
When Constantine the Great became emperor he expanded the army and bureaucracy again. Again this came coupled with new taxes and increases in old taxes. These taxes impoverished the lower classes of Roman Society, the craftsmen and minor merchants, and led to him issueing new obligations and rights to land owners. He ordered them to keep their tenants in line because these taxes were so high that they were abandoning their own land in masses. These immigrations happend primarily in the Western Empire and these new rights and obligations were the second step to serfdom.
Fast forward again, this time to the collapsing Western Empire and the transition period to the successor kingdoms. Due to the wealth differences in the east and west (and various other reasons), the eastern cities thrived and the western cities declined. Al the while Imperial authority in the west declined, allowing the landowners and army commanders to grow in influence. At some point it wasn't able to enforce the tax system on farmers and craftsmen, allowing them to leave their vocations and land, further hurting tax income. While this was going on the lower lever Aristocrats that ruled the cities and surrounding areas were being crushed by new taxes in order to make up for the taxes lost. This while they had to use their own funds to maintain the public facilities and the previously mentioned tax deposit.
Meanwhile local farmers had enough of the ever increasing taxes and organized themselves underneath the local landowners, be they big or small. These landowners supported by dicontent farmers started to ignore the local governments and because of their relative wealth and food production these landowners could afford to hire their own guards. Due to the declining situation in the cities, whom lost the economy they were dependent on, high taxes and a collapse in central authority more and more people would flee to these villae-islands where there was work, people could survive and they were safe from bandits and tax collectors. In return for their safety they often renounced their citizenship and the rights thay it gave them, which shows how willing they were to get away from the cities and their taxes. Over time this cooperation degrade into a more master-servant relation.
Cities declined further, with the few remaining rich leaving and the necessary public facilities falling in disrepair. In the end these shells would be absorbed into the administration of the local landowners or bishops. Fun fact, local resistance to the invading tribes was low because they were asking less taxes than the old Emperors and used what was left of the old Roman institutions.
The fall of the Roman system followed a very specific series of events, so I find saying that it will lead to feudalism a bit dishonest. if we end up facing a similar breakdown as the Roman state did I expect us to be even worse of than feudalism really, given how dependent every part of our society is on the state.
As people might have noticed from my last post on the Romans, generalizations surrounding their collapse are a bit of a pet peeve of mine.
Sources:
Een Kennismaking met de Oude Wereld (An introduction to the Old World), L. de Blois.
A Short History of the Middle Ages, B. Rosenwein