People were not talking about just training the sword though, but to integrate other rare/unique advantages we possess to even the field between us and opponents that have trained for much longer than us / are more dedicated to martial pursuits than us.
Branalhune is perhaps our single most powerful martial tool, so it makes sense to build much of our whole fighting style around it. In that sense, incorporating our magic (shadowsteed, teleportation, etc.), arcane marks, magic items and other attributes into our sword style is probably the best way to improve our martial abilities and perhaps reach Grandmaster Swordsmanship.
We might not have centuries/millenias of training in the sword, but we have other aces that, if integrated in our sword style, help even the score.
EDIT : Also, considering our last investments in the sword got us a completed sword style which might be one of the singular most important advances in lethality, I don't think the argument about "diminishing returns" is true. If anything, I think that there is an argument that improving our sword style brought us EXPONENTIAL returns rather than diminishing ones or even linear ones.
There is no reason not to believe that Grandmaster Swordsman isn't another gulf compared with Master Swordsman, especially if we attain it by combining all our advantages.