Let's talk about answers and their quality

Question

A long time ago the WB.SE community was discussing whether to adhere to the Back It Up! policy or not. Those discussions are not very relevant today since most participants are long gone now. And the current generation of users has very different standards compared to the early days of the WB.SE. I think that it is time to talk about answers and their quality again.

The WB.SE tour says: 'With your help, we're working together to build a library of detailed answers to every question about worldbuilding.'

I believe it is an admirable goal. But it does not say whether this library should contain answers with good, reliable information or it can be made of speculations and wild guesses. The Help Centre is also not very specific:

Read the question carefully. What, specifically, is the question asking for? Make sure your answer provides that – or a viable alternative. The answer can be “don’t do that”, but it should also include “try this instead”. Any answer that gets the asker going in the right direction is helpful, but do try to mention any limitations, assumptions or simplifications in your answer. Brevity is acceptable, but fuller explanations are better.

As it is now there is no formal rule addressing the quality of the answers. There are also no guidelines for answers, except for those stated in Good Subjective, Bad Subjective, but it is a rule that is systematically ignored on the current WB.SE (some high reputation and active on Meta users do not even know about its existence).

Today's WB.SE has a great number of answers that appear to me as poorly researched, based on speculations, unverified assumptions, personal preferences, 'common sense', ideologies, tropes, and so on¹. As a result, it is not possible for me to trust the answers (not to mention that TV Tropes is a much better source for tropes). Their best use is to test the audience, their preferences, values, and beliefs.

Speculative answers and wild guesses are not bad per se. And they can be immensely helpful. Even answers based on wrong assumptions can be helpful if those assumptions are clearly stated. The problem is that they do not work well with the existing requirements for questions. The WB.SE does not welcome speculative questions and requires prior research. There is also a rule against opinion-based questions, but at this point of the WB.SE history, it is not clear what constitutes opinion-based.

I do not have a clear position on this. I would like the expectations for answers and questions to match each other (researched questions and researched answers, for example) because I believe in reciprocity and it seems to me that it would be easier to keep the quality of the questions high if the answers are of equally high quality. However, I do not expect the community to support this view or the WB.SE to uphold it. At this time I am trying to learn what other people think about the answers and whether there should be some guidelines for them.

---

Clarification:

The purpose of this query is not to start a debate about my own position regarding the quality of answers or my evaluation of answers. The former is not possible as I do not have a clear position (which I stated before) and I am not proposing any policies or guidelines. The latter is something that I wanted to avoid because the discussion will turn to be about specific answers and their merits rather than the general quality of answers and whether any guidelines are needed.

My intent was expressed in the last sentence of the original version: At this time I am trying to learn what other people think about the answers and whether there should be some guidelines for them. In other words, I wanted people to share their thoughts about answers and their quality and whether any guidelines are necessary.

---

Since @elemtilas insists on providing specific examples of what I would consider a low-quality answer and explicitly states that they would want to hear my opinion about their own answer, I will use their answer to this query (as of 4 February 2022, 18:20 GMT) as such an example.

There are 2 big problems with this answer:

1. it is a response to the linked Back It Up! policy discussion, not this query (and @elemtilas does acknowledge this)

   It is worth noting that @elemtilas justifies this approach by saying that my query is unfinished and not clear enough to them. They even threatened to VTC it.

   It should be mentioned that @elemtilas is the author of Is it Fair to Encourage People to NOT Answer Questions that Need to be Closed?, where they advocate for not answering questions that they find to be unsatisfactory. Therefore, their own response is not only inconsistent with the WB.SE guidelines for answers but also with their own position regarding answering questions.

2. in their response @elemtilas does not represent the original Back It Up! policy or discussion about it accurately (and they agree with this evaluation)

   The way @elemtilas builds their argument and formats their post distorts the original meaning of the Back It Up! policy and may lead readers to believe that it was an attempt to require references for every single answer. It was never the case. The core of the policy was: Every answer needs to justify the conclusions it comes to.

I see this kind of answer not only as poor-quality but detrimental to the WB.SE. It sets a precedent for being off-topic. It also misleads the readers. In addition, it makes it impossible for me to trust the author's words even when it comes to their expressed attitudes and positions on issues because their acts are not consistent with their words.

I will not provide any additional specific examples of low-quality answers as I do not believe that it would be very constructive to this discussion. However, since @elemtilas has difficulty understanding my point about disaster myths I will add a note on this.

Every answer that features mass panic or social chaos as inevitable and 'natural' (as in 'it's human nature') outcomes in disaster scenarios is an answer that lacks research because both of these notions are among debunked disaster myths. There is even a Wikipedia article about this. Moreover, these answers are, if I put it kindly, irresponsible since the propagation of disaster myths worsens the outcomes for real people affected by real disasters.

---

^{1 My personal favourites are mass-panic and social chaos. These myths were debunked so many times (please google for 'disaster myths'), yet, almost every question related to disastrous scenarios will have answers suggesting that mass-panic, looting, rioting, and social collapse are inevitable.}

^{This lack of research is not limited to social phenomena, it shows in answers to questions that are related to natural sciences as well.}

Answer 1

Our Stack is not and should not be Physics.SE-lite

• How do you cite or document an answer about a user's magic system? Or the magical creature they're designing? Or the magical weapon they're designing? This is the fundamental problem with the "Opinion Based" closure reason — it's not possible to become an expert in Magic from the perspective originally intended by Stack Exchange. This is especially a problem when a querent, who ostensibly knows their own magic system better than anyone else, gets an answer from someone who is familiar with a dozen magic systems used in various RPG games and/or novels. Which one is the "expert?"

• How much documentation or citation is deemed satisfactory when answering questions about dystopian governments? post-zombie-apocalypse societies? or time travel? All of these can draw from (and would be benefited by) current mathematics or historical conditions that reasonably represent the goal of the OP. But so would drawing on years of reading time-travel, post-apocalypse, and dystopian-government novels as what other authors have done is a perspective as valid as any science-based or history-based answer. But should we ignore the perspective of a user who grew up, for example, in East L.A. and has a unique perspective of what "post apocalypse" might mean, despite a complete lack of credentials?

And that's the problem. In my opinion, we already focus way, way, way too much on being a science-only Q&A site. We're robbing the site of the one thing that really makes it valuable: imagination.

• For my last example, how do you document the behavioral responses of an artificial intelligence? or a government? or a military contingent based on a provided context or circumstance? How do you design the sexuality of an alien species that is unrelated to Humanity or any terrestrial creature? Questions like these that are not well bounded are quickly closed, but we've answered those that are well bounded and don't reflect the Help Center restriction that we don't answer questions about character choices. You, yourself, have noted in a previous Meta post that questions like these should either be left to the experts to people who take the time to research the issues such that an expert in the field would not be disappointed by the result or excluded entirely rather than risk "falsehoods."¹ If we held all questions to that standard, the only people answering would be accredited experts in their fields (yeah... magic...) or no questions would ever be allowed on the site.

How do we improve answers knowing that all kinds of people will use this site?

I once had a comment-discussion with an advanced user over at English.SE. He was constantly trying to push questions from English-as-a-second-language users and your-question-is-too-basic users off the site. When I pointed out to him that there wasn't actually a practical way to do that without violating SE's CoC, his response was, "can't we professionals have a site for ourselves?" The answer, since SE doesn't require CVs as part of the account creation process, is "no."

Stack Exchange must be for more than "professionals" — experts in their fields who not only have been educated well enough to provide authoritative answers but who also have the time to sit down and write thorough, documented answers. I'm only guessing, but I suspect a substantial portion of our users haven't finished (and may not finish) college — and there's a chunk who haven't finished high school. Expecting them to not answer before they've finished their training is unrealistic.

Further, we have a Help Center page dedicated to discussing how to answer a question. The bullet I think should be at the top of the list is "answer well-asked questions," which many users ignore (if they know the rule at all). Holding querents to well-asked questions would do more for improving answers than hoping any new user will improve their answering skills. Here's that page for reference:

• How do I write a good answer?

Besides, the whole point of the up-vote/down-vote system is to express the belief that, in the case of a question:

(UP) This question shows research effort; it is useful and clear.
(DOWN) This question does not show any research effort; it is unclear or not useful.

and in the case of an answer:

(UP) This answer is useful.
(DOWN) This answer is not useful.

That the average user uses up-voting/down-voting to merely express popularity or appreciation is beside the point. A mechanism exists to filter poor answers. (And if they're really poor, they can be flagged for deletion).

So, what do we do when, for example, an answer expresses an academically out-dated belief that's still popularly regarded as true?

Unsurprisingly, this is NOT unique to the social sciences. Humanity's understanding of both themselves and the universe they live in is constantly expanding and evolving. New hypotheses replace old hypotheses and it takes time for that to percolate through the general populace.² I've had several answers over the years where people pointed out to me that my understanding was outdated.³

But does that make my answer useless to the querent?

And that's the fundamental problem with defining "answer quality" on a site that's dedicated to imagination and creativity. The casual insight of an "uneducated" user can quite literally be the exact answer the OP was looking for because we're supposed to be focused on building fictional worlds, not copies of Earth.

@Python's right, we should allow the querent the privilege of caring about what a "quality answer" looks like.

We have tags that theoretically set these standards.

• No tag..., useful answers are open to the interpretation of the respondents based on the expectations set in the query and the tags used to identify its context.
• magic, anything goes so long as it operates within the querent's magic system.
• science-fiction, indicating that while science should be the basis of an answer, wide birth is allowed in its interpretation or application, even to fantastic results.
• science-based, indicating that science should be the basis of an answer, and how it's applied should represent at least a suspension-of-disbelief scientific application, but outright proving your point with mathematics or citations is not required (but appreciated).
• hard-science, indicating the answer MUST be the basis of an answer, that it's application MUST adhere to a strict interpretation of science, and the answer MUST contain supporting mathematics and citations.

I've complained before about the science-* tags because they don't cover the gamut of querent expectations and don't entirely express what a good answer would look like.1,2,3 But there is a LOT of historical inertia behind those three tags, so they're really difficult to improve.

Nevertheless, they (combined with the Help Center page and the "Good/Bad Subjective" blog) represent the Stack's effort to achieve your goals. Although I believe improving the questions will go further toward improving answer quality than will educating users about writing good answers, I'm all for trying to do it. But...

Here's the problems, based on experience.

• Users almost never read tag wikis. And even when they do, they frequently don't use tags properly (the reality-check tag is the worst offender and best example). Worse, because we don't even mention the magic, science-fiction, science-based, and hard-science tags in the Help Center, new users don't know they exist until they see them on other posts. It's amazing how many times new users use the hard-science tag thinking it refers to the literary genre.

• If it ain't in the Help Center, it doesn't matter.

• If it is in the Help Center, it's an uphill battle to convince anyone to do it.

• No matter how thoroughly or clearly the expectations are written, there will always be someone who thinks the person asking for a better answer is misinterpreting/misapplying the expectations or violating the CoC for pointing out someone else's weaknesses.

And finally...

• I think a concise explanation of answer quality expectations that meets the needs for the wide variety of topics we allow on this Stack would be next to impossible to achieve. But that's just my opinion.

Conclusion

I think there are other issues more important than this one.

---

_{¹ And here you and I will likely always disagree. I don't consider Electronics to be a definitive science. Our understanding of it changes. There are things twenty years ago we EEs knew from mathematics and experience could NEVER be done. And yet EEs are doing them today. Social sciences don't even rank on the same scale of predictability — and yet you want us to treat them like they do as if today's understanding of human behavior, like today's understanding of Electronics, is somehow the end-all of knowledge and not just the current best-guess. I get it that this Stack would benefit from a better understanding of today's social and behavioral hypotheses... but I'm not willing to take them more seriously than I do my own academic background.}

_{² A fair number of people still think oil comes from dinosaurs.... Although I'd hope the majority of said believers are over the age of 20. If not, schools must really stink.}

_{³ The majority of times this has happened to me have been questions about Einsteinian Physics and Climate. The last time I cracked a book about those subjects was 30 years ago. Back then, those textbooks were considered authoritative. See the problem? I had no reason to believe my answers were inadequate.}

Answer 2

First and foremost it doesn't have to be just a library, there's google scholar for that and also almost every city out there has a public library with free books.

Worldbuilding.SE seems more like a user to user website, which means question makers are free to specify what they seek in an answer and the level of opinion vs reality they are willing to accept.

Answer 3

Tag it

Loosen up for openers and answerers, but register: let tags indicate quality

My proposal: peer-mods should actively maintain tags. When SE would ever want to adhere to your above quality proposal, they could move implausible scenario questions and/or non-backed up answers into another forum section, named WB fantasy, using the tags to filter things. Your preference will be the remaining forum, that is WB scientific

SE decides, some time in the future

You won't have to bother current WB unified members with any demands. When we maintain the tags properly, SE may manage things differently and split up WB, let mods decide moves..

Freedom

Until SE splits things up, members on WB unified should be free to ask science related stuff and also things about fantasy weapons, implausible or hybrid creatures, flat Earths, or hairstyles for astronauts.. askers can acknowledge implausible answers, or accept answers without any references. When the asker doesn't get the references, peer mods can decide afterward, to remove the high quality indicators (=science tags)

1. Adding tags

It rarely happens, but anyone having 2000 points or more can add tags. You can add a tag, say science-based, when the question and answer are sound and references are put. Or when the question asks for a plausible and realistic scenario and it has an acknowledged answer, add a reality-check tag.

2. Removing tags

Remove a tag, when a question, or its acknowledged answer does not meet requirements of the tag: there should be proper "Back it Up", when there is a science-related tag involved, also in the question there should be references.

Answer 4

Dipping My Toes in the Fuming Acid Bath

EDIT: This response addresses only the original, unedited question.

I do have some comments on the linked discussion about a proposed Back It Up policy.

Mourdos: By in large I concur with this answer most strongly. WB is actually designed to handle good quality speculative and opinion based queries. (I think we've got enough consensus to say that a query like "which colour should my fairy godmother's wand star be? I like green, but I want all your opinions!" is doubleplus ungood ad extremissima ipsissima. Almost every other query in this forum, barring the hard science and science based questions are to some extent looking for an opinion.

I have no problem with divulging sources if I think they're really and truly warranted. But on the other wing, I too believe that insisting upon sources for speculation and imagination will not only be detrimental to this site, but would probably kill it.

Community Wiki: This one is really an answer to different question, but I really do like the idea of a Pseudoscience / Fantastic Science tag that would compliment the hard science / science based tags. Like the science tags, the pseudoscience tags would require some research and sources; only in stead of physics and chemistry, we'd be looking for standard works on alchemy, magic, divination, homeopathy, chiropractic, theosophy and discarded science (phenomena like phlogiston or the expanding Earth theory).

Such tags could reduce the amount of wild speculation and zany opinionification that questions of magic sometimes engender.

KRyan: The OP's own response addresses quality fairly well, and I'd consider it the "common sense" response. You can give the zambonniest answer ever seen on WB, but so long as it ticks one of these "back it up" boxes, I'd likely consider it a quality answer.

Styphon: I 100% concur with the notion that encouraging respondents in general to link to their sources, we shouldn't be in the business of enforcing such a rule. With the exception of hard science / pseudoscience, which do / would require citation.

I'd say that the overall solution is to encourage users to back up their facts wherever absolutely necessary, but not to enforce such a policy except in the case of tags that require citation. This I think is the natural extension & compliment to Mourdos's answer.

Slathering on the Caustic Lye

Once we have some linked examples, I'd like to explore Otkin's underlying questions.

Answer 5

WtF

The query has now been edited to include a whole host of issues ranging from the OP's evaluation of my answer to the OP's psychological profile.

This response only addresses the newly added section.

The purpose of this query is not to start a debate about my own position regarding the quality of answers or my evaluation of answers. The former is not possible as I do not have a clear position (which I stated before) and I am not proposing any policies or guidelines. The latter is something that I wanted to avoid because the discussion will turn to be about specific answers and their merits rather than the general quality of answers and whether any guidelines are needed.

Fair enough. I can't speak for others, but I didn't read your question as if you asking us to help you evaluate answers. While it's not immediately clear what you don't have a position on, it is pretty clear what your question is about. This clarification is welcome just to ward off any evil spirits.

My intent was expressed in the last sentence of the original version: At this time I am trying to learn what other people think about the answers and whether there should be some guidelines for them. In other words, I wanted people to share their thoughts about answers and their quality and whether any guidelines are necessary.

For my part, this is what I asked you clarify. I can't reasonably talk about "what I think about the answers" unless we're looking at the same set of answers. Before fruitful discussion can begin, we need to establish common ground and review common data.

I'd still love to have this conversation with you (and others)! None of us can do this meaningfully if you refuse to show us some answers that you'd like to discuss!

Since @elemtilas insists on providing specific examples of what I would consider a low-quality answer and explicitly states that they would want to hear my opinion about their own answer, I will use their answer to this query (as of 4 February 2022, 18:20 GMT) as such an example.

Specifically, I asked about any answers I've written that fall under what you categorise as poorly researched, based on speculation, unverified assumptions, personal preferences, 'common sense', ideologies, tropes and the like. My answer was based on reading your original query, asking for clarification, reading all the answers to your question, reading the linked question and answers and all the associated comments. That's research. My answer begins with commentary on the linked discussion; it is not yet complete, as I await further clarification from the OP. I didn't make any assumptions, but when I do, I verify all of them. Preferences, common sense, ideologies and tropes didn't figure in to the answer I'm writing, and I honestly don't see that they will, though they might! If they do, I will note such as is customary.

I should, however, have been even clearer! I read the question as referring to quality of answers on Main, not here in Meta. I've written 455 responses on Main. Perhaps you could evaluate a few of those?

Especially within the context of my other Meta answer being a WIP contingent on you clarifying your query!

There are 2 big problems with this answer:

it is a response to the linked Back It Up! policy discussion, not this query (and @elemtilas does acknowledge this)

It is worth noting that @elemtilas justifies this approach by saying that my query is unfinished and not clear enough to them. They even threatened to VTC it.

I do not "justify" anything. I was clear in stating that my initial comments were in regard to the "discussion about" the proposed policy, not about the policy itself. I offered these preliminary comments on the linked policy discussion as that post and its answers are pertinent to the OP's present query. I continue to be more than happy to address OP's main query when it's eventually clarified!

It should be mentioned that @elemtilas is the author of Is it Fair to Encourage People to NOT Answer Questions that Need to be Closed?, where they advocate for not answering questions that they find to be unsatisfactory. Therefore, their own response is not only inconsistent with the WB.SE guidelines for answers but also with their own position regarding answering questions.

The OP's question does not need to be closed. It is not unsatisfactory. There is no need assume malice.

in their response @elemtilas does not represent the original Back It Up! policy or discussion about it accurately (and they agree with this evaluation)

I do not claim to represent anything. The Back It Up! policy was introduced by the OP in the context of a query about the state of present time answers. It is thus a valid topic for discussion within the context of any answer to the OP's query.

The way @elemtilas builds their argument and formats their post distorts the original meaning of the Back It Up! policy and may lead readers to believe that it was an attempt to require references for every single answer. It was never the case. The core of the policy was: Every answer needs to justify the conclusions it comes to.

I did not "build an argument" for anything. I structured my response to separate one section of comments on the responses to the linked policy from a WIP section on the question at hand. I have no comment on the original policy at all. If I did, I would either respond to it directly or else draw up a revised proposal of my own. I have no reason to do either.

I see this kind of answer not only as poor-quality but detrimental to the WB.SE. It sets a precedent for being off-topic. It also misleads the readers. In addition, it makes it impossible for me to trust the author's words even when it comes to their expressed attitudes and positions on issues because their acts are not consistent with their words.

It is poor quality only in so far is it remains incomplete. It remains incomplete until the question is finally clarified. Once the question is clarified, I guarantee you'll get a stellar answer that not only addresses the linked policy discussion, but also addresses any issue you have with the state of more recent answers in the forum!

The linked policy and its responses are central to the present question. Just FYI: you have also brought your own inner psychological issues into the question, among which appear to be trust issues, belligerence, history of accusative behaviour, passive aggression and so forth. Otkin, do you really want to go down this road? I strongly urge you to purge your question of any of your personal issues.

I will not provide any additional specific examples of low-quality answers as I do not believe that it would be very constructive to this discussion. However, since @elemtilas has difficulty understanding my point about disaster myths I will add a note on this.

Fair enough. You've just determined that your own question is now pretty much unanswerable. I think it could be argued that this is a valid Vote to Close rationale.

Every answer that features mass panic or social chaos as inevitable and 'natural' (as in 'it's human nature') outcomes in disaster scenarios is an answer that lacks research because both of these notions are among debunked disaster myths. There is even a Wikipedia article about this. Moreover, these answers are, if I put it kindly, irresponsible since the propagation of disaster myths worsens the outcomes for real people affected by real disasters.

I can only take your word for this. I honestly don't know how answers that involve mass panic or social chaos fit in with the linked policy, and since you're still not "backing up" your assertion with any kind of data, I'm just going to have to leave it at that.