Well, it depends on what exactly are the mechanics of the whole retaining memories.
If you want for Destroying souls to not be more immoral than killing someone, you can either make so that memories are either stored elsewhere and then randomly linked to another soul, or that the bits of soul that keep memories are far more hardy than the rest of it, thus allowing them to escape complete destruction.(And then go infective/cancerous and infect other souls with mismatched memories. Or become a ghostly abomination in the underworld.)
Or something else entirely. I am not proposing and/or developing an Alternate Universe, i am merely asking questions about the implications of something.
Oh, no. I'm just speculating trying to spark discussion as well. It's interesting to me cause it actually came up in a game I'm in recently.
My character is planning to learn Necromany soon and in the rewrite of Sorcery we are using, 'Unconquerable Self' is part of the basic Sorcerer package. Unconquerable Self instantly sends you soul dirrectly to Lethe/scrubs all your memories from your Exaltation(and a few other things). The Necromancy version is almost identical except it send your soul directly to Oblivion.
I commented my character, if she knew she was going to die, would likely use it because while she doesn't believe in 'reincarnation' (Which is to say, to her, saying 'the hun after passing through lethe moves on to be inside someone else' is equivalent to saying "Atoms that were inside stars/Hitler/Albert Einstein are inside you", a true fact but meaningless to her) the thought of her soul being inside someone else is a little weird to her and presented with the option for that not to be so, she'd take it.
Now other characters actually do believe in the continuity of identity between reincarnation, and one of the other character's player commented if his character ever heard that, the character would very likely consider killing my character before they could so as to 'save' all future incarnation of her. ((Note: He also stated his character would never ACTUALLY do that for the sake of the game. ))
And I think that's a real interesting dichotomy. We have setting that's very clear reincarnation is real, but it's left super vague to what extent there is some, if any continuity of identity. Even if you make it so there is some level of memory retention between all reincarnation, there still is the question to what extent does that mean someone is 'like' someone else. To what extent will they share predispositions and personality traits, does it vary, is it completely random? Especially in a situation where access is, while innate, still a consciously activated skill... It's certainly interesting. In that case it's objective that if you destroy your soul you're depriving the world the benefit to the access to your life expereinces, making the destruction of a soul certainly 'selfish' to 'dickish' even on the best end.