Your grandfather was not only charismatic but also genuinely enjoyed being with his partners and making sure they enjoyed themselves too. He was also a widower. Most of his peers were of the more traditional mindset about how things between husbands and wives worked and only cared that they had an heir and a spare before finding other amusements. So you have a lot of unsatisfied wives with husbands that were aloof at best. And a charming man who lost the woman that he loved dearly and being open to comfort. As for who he pissed off... well you were never given the full list, but you know that if your grandfather had ever set foot back in Heaven's Peak he likely would have been hung. And yes, at least one woman in the crowd would have made the obvious joke at his sentencing. There is also the possibility that the Church considered his conduct to be an embarrassment to one of the Holy Cities of the Kingdom of Light, and as such cooperated with his enemies to get him removed to Spree.