There's no need for a magical explanation for why sealing seems to make sealmasters crazy. I propose three non-magical effects working together.
First, people who choose to continue studying sealing after the risks have been made sufficiently clear to them are probably at least a little bit cracked to begin with, so you've got an initial selection bias going on.
Second, studying and practicing sealing is an exercise in constant low level (or occasionally high level) mortal terror that something will go horribly, horribly wrong. It's the sort of stress you usually see in characters in Lovecraftian stories. To put it another way, seal masters pretty much all get PTSD/anxiety disorders because of the sheer psychological stress of the activity.
Third, the sealmasters who survive long are going to be the most paranoid and obsessive ones, so there's a survival bias going on slanting the sealmasters people encounter toward the paranoid and obsessive.
Jiraiya seems interesting in that he's potentially a notable exception. Somehow he managed to not let the stress get to him or turn him paranoid and obsessive, and he wasn't too cracked to start with.