Despite the traditional roguish image, dual-wielding does not work very well for Rogues, because neither of their two most tempting class features – Sneak Attack and Cunning Action – work with it. Sneak Attack only works on one attack per round, and so does not scale with dual-wielding. Cunning Action gives you new ways to use your bonus action, which dual-wielding occupies.
Similarly, dual-wielding does not work hugely well for Barbarians. Many of their features (Rage, Reckless Attack) rely on Strength, and dual-wieldable weapons are more suited to Dexterity builds due to the common link between the Finesse and Light tags. This can be mitigated with a feat, but that same feat can be spent bumping your way up to Strength 24 or buying a greatweapon feat – and it invalidates Sneak Attack, anyway. Moreover, the higher end of Barbarian damage-dealing relies on combining Reckless Attack with Brutal Critical, which requires a Greataxe or similar d12 damage weapon to get the most out of. Also, you can't dual-wield on a round that you enter Rage.
Hell, dual-wielding probably isn't a hugely good investment for a Fighter, either, most times... In fact, Improved Divine Smite, no bonus action features and only one Extra Attack means that the Paladin's probably the relative best class for dual-wielding, ironically enough – even without the relevant Fighting Style.
Of course, you're aware of all this. My usual distaste for multiclassing aside, I'll just say that (if my math is right) your build with the relevant feat will be doing
roughly the same per-turn damage as a Barbarian 17/Fighter 3 (Berserker/Champion)... just without so much swing, because the Barbarian 17/Fighter 3 is making 6-8 rolls a turn where they crit on a 19-20, re-roll damage dice on a 1-2, can use their bonus action to make another attack if they crit or kill, and can take an attack penalty for a damage bonus, and taken all together that does
weird things to the damage average.
All that said, Battle Master is still probably better bet for the build if it's crit-focused, because you can spend your superiority dice to add damage
after you've confirmed a crit.
So it looks totally viable, though you've not stumbled onto anything broken. It seems like you'll mainly just be keeping up but also having more options, which is always fun.