I pretty much agree with what HDE already wrote. However, I want to specifically note two particular parts of your question:
some of my responses (I feel) should be directed to the individual (rather than made public) because some of the debate is about the appropriateness of the answer or question, and in fact whether or not I should delete or edit.
It seems to me that what you are basically saying here is that you feel that most people aren't worthy of being privy to your reasoning why a post is good (or bad, as the case may be).
That seems like a dangerous way to run things.
Part of what makes Stack Exchange so successful, and so good at filtering the good answers from the bad ones, is the community moderation. Each vote on each post has limited influence, but taken together, this actually does work pretty well to separate the wheat from the chaff.
If arguments are held in private, even if those arguments are completely civil (which, as HDE discusses, they might not be), then others have less information by which to make their judgements on a post. That is, at best, not good.
Diamond moderator tasks on all but the largest sites in the network is pretty low-intensity, in that there's usually little that specifically a diamond moderator has to do. Most things can be handled by the community. (That, too, is how things are generally handled on Worldbuilding; most moderation activities that can be handled by the community actually are handled by the community, not by the diamond moderators.) If you add something to the system which is not accessible to the community at large, now everything that happens using that feature must be monitored by the diamond moderators to ensure the feature is not abused. This would cause moderator workload to spike. At best, such as with a private messaging system, perhaps only flagged content would need to be reviewed; but even if so, this would still cause an increase in moderator workload, for what appears to me to be dubious gain.
Editing answers / questions makes the debate public, and that's not always the best (especially for side issues that are only tangentially related to the OP).
If it's a side issue only tangentially related to the question, then why are you including it in an answer in the first place?
A wise person is supposed to have once quipped "a letter is perfect not when there is nothing more to add, but when there is nothing more to be taken away". In other words, each and every word serves a purpose in communicating relevant information. While Worldbuilding SE is fairly lax, I still think that's a reasonable goal to strive for: include the relevant information, and omit the useless fluff. The line separating the two can sometimes be difficult to identify, but it's usually fairly easy to tell if something falls well to either side of that line.
If a major part of your answer is about something not raised by the person asking the question, then consider whether it belongs there in the first place. Maybe it's better off as a different question and answer (maybe on a different site in the network), or on your personal web site or blog, or as an article in a peer-reviewed journal (you wish!), or on a regular discussion forum, or if it's better to simply sack it. Nothing prevents you from linking to external content, as long as the answer itself is self-contained and ideally the links are relevant. Most likely, wherever that extra set of points might belong, they're better off there than in an answer to an unrelated question.