9
$\begingroup$

I was reading a previous question on data on close votes, and a related question came to my mind.

The question is as follows: is it possible to reject the hypothesis that there exists well-defined cliques of users that have a higher than random frequency in co-voting to close questions?

As a noob in terms of community dynamics, my expectation would be that the specific sample of users that vote to close any given question should be a random sample of the the population of users that in general cast a vote to close questions.

If that is not the case, I am of the opinion that there could be other factors at play, some of which should be considered and addressed as a community. For instance, there could be cascade effects, such that a user vote to close prompts a fellow user (let's say a friend) to vote to close, regardless of the merit of the question per se; or, it could be topic driven, i.e. some groups of users do not like (otherwise reasonable) questions on certain topics, hence the co-voting patterns; or, it could be that there is a group strategy (explicit or just emergent) to assert dominance and control within the larger group of the community. You name it.

I would tag this question as hard-statistics, in case.

$\endgroup$
  • 3
    $\begingroup$ Surely your starting hypothesis should be the null hypothesis (there exist no well-defined cliques of users that have a higher than random frequency in co-voting to close questions) - the other way round requires you to prove a negative, which is generally impossible. $\endgroup$ – walrus Dec 7 '17 at 12:51
  • $\begingroup$ I would be quite surprised if a mod couldn't whip up some statistics, although they can't share the actual data if I understand this correctly, but could share the results. Another factor that could be at play is time. Users in similar time zones, or who tend to be on at the same time, would tend to co-vote more. $\endgroup$ – Gryphon - Reinstate Monica Dec 7 '17 at 13:30
  • 8
    $\begingroup$ @Gryphon Activity is also a factor. More active users are more likely to be keeping an eye on the review queues. Just because the same five (or however many) high-rep users keep voting to close doesn't mean there's a conspiracy. Those users reached their high rep for a reason. $\endgroup$ – Frostfyre Dec 7 '17 at 13:34
  • 5
    $\begingroup$ That's true. It could also be that certain users are more knowledgeable about certain topics. Therefore, they tend to know that a question on that topic requires more information or is too broad, better than other users. Therefore, they would tend to close more questions in those areas, which could also skew the statistics. $\endgroup$ – Gryphon - Reinstate Monica Dec 7 '17 at 13:37
  • $\begingroup$ Moderators have access to data that ordinary users don't. In general, such data cannot be disclosed in detail; the general rule of thumb being unless the same conclusions could be reached from data that is visible to ordinary (perhaps high-reputation) users. (The moderator agreement does to some extent govern this, and is publicly visible.) However, with very few exceptions (for example, it is at times possible to infer this from comments, reputation changes, etc.), moderators cannot see who voted which way on a post. $\endgroup$ – a CVn Dec 7 '17 at 13:39
  • $\begingroup$ On the whole, I'd say Frostfyre, and Gryphon about subject matter expertise, are likely to be correct. Those are certainly more plausible hypothesis in my mind. $\endgroup$ – a CVn Dec 7 '17 at 13:42
  • 5
    $\begingroup$ It would be possible to use SEDE to analyze close votes that ended in a closure (and likewise for reopen), but votes that aged away or were retracted or just never reached critical mass aren't in the public data. Moderators can't query that either. $\endgroup$ – Monica Cellio Dec 7 '17 at 14:03
  • 2
    $\begingroup$ @MonicaCellio Thank you for the tip to look into SEDE. I had never looked at it before, but it turns out it is just a SQL database, which also happens to be what I do for a living. Now I have access to everything, and he who has the information has the power, muahahahah!!!.....sorry where was I..... Thanks for tipping me about SEDE! $\endgroup$ – kingledion Dec 8 '17 at 14:00
  • 1
    $\begingroup$ @kingledion I look forward to seeing what else you do with SEDE in the future. :-) $\endgroup$ – Monica Cellio Dec 8 '17 at 15:07
  • 2
    $\begingroup$ One note I would add. I have often, and moreso now that I am a mod, posted questions I believe should be closed in chat. There is only a tiny fraction of the site's active users in chat so I have no doubt that common themes in users vtc'ing exist, I don't know that I see that as a problem as its never been a coordinated, nefarious effort. $\endgroup$ – James Dec 11 '17 at 17:49
  • $\begingroup$ @MichaelKjörling Plausible hypotheses are best supported by evidence. $\endgroup$ – a4android Dec 12 '17 at 5:10
  • $\begingroup$ @James emergent group strategies need not be coordinated, nor they need to have a nefarious purpose in the mind of the individual. They simply emerge as a global pattern from the isolated choices of multiple actors. As a group strategy they can result in an undesired outcome, though. $\endgroup$ – NofP Dec 12 '17 at 8:42
  • $\begingroup$ All true. I was just noting a potential source. $\endgroup$ – James Dec 13 '17 at 3:16
16
$\begingroup$

Data

I analyzed all the data for 2017. I got the list of question-closers for 1719 questions that have been closed for any reason but not deleted (deleted posts usually have no upvotes or answers after 30 days). This also counts questions that have since been reopened. I then took every subset of these closers and counted them. Here are the results

Groups of 1

The most closing user voted on 907 close questions, or 53% of the total. There are 11 users who voted on 10% or more of the close questions and those 11 users placed 2.9 votes per closed question. Or, in other words, they provided 59 % of all close votes.

Groups of 2

The highest grouping of 2 voted to close 569 questions together, or 33% of all questions. There were 10 pairs of users from 6 individual users who voted on 10% or more of closed questions together, and they placed 1.6 votes per question, or 33 % of the total close votes.

Groups of 3 or more

The highest grouping of 3 users voted to close 187 questions together. The highest group of 4 was 48 questions, and the highest group of 5, only 7 questions.

Gini Index

The Gini index of close voters is 0.807. Gini is a measure of inequality, used for incomes mostly. I don't have any information to put this number in context, but I am interested in doing so in the future, by comparison over time, or with other sites.

Pearsons $\chi^2$ test for independence

Forming the the 11 highest users into a diagonal contingency table, we can use Pearsons test for independence. Using the null hypothesis of independence, I built a contingency table with 50 degrees of freedom. That table had a Pearson's $\chi^2$ score of 5492, which is very high. Since $p = 0$ for this test, we can (resoundingly) reject the null hypothesis.

A cursory look at the data shows us why.

Some users close a lot of questions

Especially recently, there are two users who have closed 'most' of the questions. Since August 1, there have been 718 questions closed; these two users closed 75% and 73% respectively.

These two users have voted together 399 times, or 76% and 74% of those users total close votes, respectively. Together, they have voted to close 56% of all questions.

There is a third user who frequently votes along side these two. This third user has 207 close votes, of which 57% have been with both of the two high close voters, and 93% have been with one or the other.

This appears to be evidence of some 'collusion.' However, when comparing the actual number of times users have closed questions against expected, even in these recent months when many questions have been closed, they are not operating together more than expected. Here are actual and expected numbers for users A, B, and C from above:

Users   Expected Closes   Actual Closes
A + B              393              399
A + C              157              159
B + C              152              152
A, B, C            113              119

In fact, the expected values are surprisingly accurate.

If we redo the $\chi^2$ test for this time period, we see that the score has dropped all the way down to 138. This is still low enough to reject the null hypothesis ($p=0.008$), but it is much closer than before.

Conclusion

It does look like certain groups are closing questions together a lot as a clique. However, this behavior is expected due to how frequently members of that group are closing questions in general. Despite a failure of independence test for the entire year, when broken down into smaller samples, the test looks a lot better.

Therefore, we can reject the null hypothesis that close voters are acting independently, but we can demonstrate some evidence that time is the factor causing this result. Overall, I conclude that there is insufficient evidence of voters demonstrating any of the co-voting patterns that you mention in the question.

$\endgroup$
  • 2
    $\begingroup$ Thank you for this interesting analysis! I don't know enough statistics to make specific comments. $\endgroup$ – Monica Cellio Dec 8 '17 at 2:35
  • 1
    $\begingroup$ What's really interesting about this is that the most common variable seems to be the time of close-vote (if you take a short period into account, the correlation drops). Do you have a way to see if tag or close-reason has an impact? $\endgroup$ – PatJ Dec 8 '17 at 6:04
  • 1
    $\begingroup$ Wow, great analysis, thank you very much! Seems like the "dramatically different levels of activity" I expected are even more dramatic than I thought. $\endgroup$ – Sec SE - clear Monica's name Dec 8 '17 at 8:08
  • $\begingroup$ Great analysis! $\endgroup$ – NofP Dec 8 '17 at 8:40
  • 2
    $\begingroup$ @PatJ I can do both of those! Give me some time though. $\endgroup$ – kingledion Dec 8 '17 at 13:56
  • 1
    $\begingroup$ The clique proposition seems unlikely. However, this data seems to show there is a cohort of high close voting users. The high frequency of close voting seems quite sufficient to bias votes to close more frequently. If it only take five votes to close a question and there are three high frequency VTCers, then this could easily distort voting outcomes especially this is coupled with other low frequency VTCers who hold similar views. $\endgroup$ – a4android Dec 12 '17 at 5:25
  • $\begingroup$ @Secespitus This analysis confirms what I had suspected, and frankly didn't was true. Granted that even an analysis like this only suggests what might be the case. $\endgroup$ – a4android Dec 12 '17 at 5:27
  • $\begingroup$ I feel like this data doesn't quite provide for a plausible analysis. It doesn't include the close votes that didn't cause a closed question, nor does it include closed questions that ended up deleted. If there were a way to include that data, I think the analysis would be much more.. complete. Also, out of curiosity, do you remember if I was part of the top closing users? $\endgroup$ – Aify Feb 28 '18 at 7:30
8
$\begingroup$

I don't have any data at hand, but here's my feeling based on what I've seen on the site and a few things to keep in mind for this topic.

You can't see who voted to close before the closing is complete, so long as people don't explicitly comment.

If I see a question in the review queue and no comments indicating any problems I often leave a Reminder to Close-Voters to prompt others to state what they think is wrong with the question. Other people do this regularly, too, if there hasn't been any comment. So the second or third person to get a close vote in the review queue can often see at least one person who said something that may be wrong with a question.

But still: you normally don't know who voted to close until the question is put on hold. Just like you can't normally see who voted to reopen until the question is reopened.

You could go to your review history after casting your vote and see who voted for what, but that's already after the fact.

This makes it hard for cliques to know when their voting-buddy voted for something.

There is no private messaging integrated into the SE network.

Nobody can send a private message to someone indicating something like "Hey, how about we close this question?" inside of the normal network.

It would require the participants to get acquainted outside of the network, which increases the amount of work slightly that would be required, just for this kind of action.

The closest you can get inside of the normal SE network is the chat, which everybody can access to write who has at least 20 reputation. And everybody can read it, no matter the reputation. This is a very low mark and there are no other requirements to be part of this "clique". In fact we would love if more people regularly visited the chat to talk about their ideas/projects/... I've seen it sometimes (maybe once a month or every two months?) that someone posts a blatantly off-topic question there to start a discussion with the regulars who are online at the moment. Or about a question that needs reopening, because the author edited his question. Or about asking people to help with editing to give a new user some hints. A lot of the regulars in the chat have enough reputation to vote for close/reopen/edit/..., which makes this the ideal place to talk about pro and con of putting something on hold.

There was even a special chat room for talking about reopening.

Bottom line: the chat may be somewhat active at times, but there is not even enough attention to maintain a chatroom for this topic. The special room was frozen and it's very rare that a question pops up in the chat for VTC/VTO. And the discussions tend to be quite diverse. The people there are not thinking "Ah, I know that username, so the question he mentioned must be closed immediately." - we openly talk about why a question should be put on hold or reopened if the case arises. Still, you could call it a "clique" if you wanted to. It's the closest thing you can get to a "clique" on this site anyway as the amount of regular chat users is quite limited, although everyone with a very little amount of 20 reputation can be part of this very open "clique".

There are Meta discussions.

More often than in the chat someone posts something about closing/reopening on Meta. This leads to the regular Meta users seeing the question and acting on them, as a lot of them also have higher rep. Again, this could be seen as a "clique", even if it's open to everyone. It's commonly referred to as the Meta Effect. But these Meta discussions are still not something that leads to people thinking "Ah, yes, gotta close". They are open discussions where different people offer different views.

People can post on Meta with 5 reputation. Again, reading is possible even for people that are not logged in to the site. It's open for everyone. And 5 reputation is a single answer or question upvote. It's a tenth of the reputation you need to be able to comment everywhere. This could be seen as a "clique", but just like the chat, it's a very open and welcoming "clique". More activity on Meta would also be a great thing. There can't be too many people active in the chat / on Meta, so getting more opinions about how closing/reopening/... should work on the site would be great and people listen to arguments - not to the username that posted something.

Regular close voters often have somewhat similar patterns.

I know that there are a handful of people that very often vote similar to how I am voting. Seeing their username in the comments does in fact draw my personal attention because I know how they comment and realized through experience that they are very good at filtering the noise or summarizing problems/possible solutions. Data would surely suggest that we are a "clique" - mainly because they just vote similar most of the time.

Just because people vote similar doesn't mean they stop thinking the moment one of them posts something. For example those people I have in mind have a similar opinion most of the time - just not when it comes to questions tagged . There are surely similarities in the voting pattern, but also bigger differences.

A question that is put on hold without any real reason can easily be reopened.

Posting on Meta and asking for the reasons for a question that was put on hold will draw in people that look at the question and you will surely find people that see the merit. If there are enough the question can be reopened. And as you can only VTC once on every question there won't be a war. You may see patterns of people who don't like questions for example voting to clsoe these questions and people who like them and vote to reopen - in the end the whole active community decides.

People show dramatically different levels of activity when it comes to reviewing.

Some people review a lot. Some people don't like reviewing at all.

You will surely find that a few people VTC very often, especially compared with the average user. You could say that they are a "clique". It would probably look like it. But someone has to do the reviewing if we want high quality questions and answers on the site. If you feel there are too few reopen votes for example the best thing to do is to lead with a good example by raising the topic on Meta/in chat regularly and trying to focus on the reopen review queue. But just like you might want to focus on the reopen queue you will find people who focus on the closing queue. Because that's also an important part of the community.

Conclusion

I don't think there are "cliques" of people on this site. Yes, there are some people who are more strict/lenient when it comes to certain topics and there are groups that you will find more often as reviewers, such as regular Meta users and regular chat users, but it's not like they are always voting for the same thing or stop to think once they see someone vote. And there are no groups who regularly invite others to close a bunch of questions they don't like. This would probably have the adverse effect quite often - if you can't say why you want to close something I for example tend to give the querent the benefit of doubt.

But I am sure that any data would reveal that some people are very often co-voting. I can think of a few people that I would expect to see with a similar voting pattern to mine because we have a similar take on a lot of topics. Participating in a lot of discussions about these topics, because we always had them and will always have them on this site, leads to that, which is also a good sign that there are certain rules that people can follow to know whether something is on-topic or off-topic.

$\endgroup$
  • 4
    $\begingroup$ Good analysis. Also, if chat (or meta for that matter) seems like a clique, it's an open, opt-in one -- very different from those cliques we experienced in high school where the whole point was to keep people out. $\endgroup$ – Monica Cellio Dec 7 '17 at 14:01
  • $\begingroup$ @MonicaCellio Yes, I thought I made that clear, but I will edit it to be more explicit. Thanks $\endgroup$ – Sec SE - clear Monica's name Dec 7 '17 at 14:06
  • $\begingroup$ That was just amplification, not a criticism of your post. :-) $\endgroup$ – Monica Cellio Dec 7 '17 at 14:07
  • 2
    $\begingroup$ Anyone (including users not logged in) can read the discussions in chat. It's only talking in chat that requires 20 rep. I've often opened a chat room in a private/incognito browser window just to see what was going on without really entering the room. $\endgroup$ – a CVn Dec 7 '17 at 14:20
  • $\begingroup$ @MichaelKjörling Good point, I also added that to the chat/meta part. $\endgroup$ – Sec SE - clear Monica's name Dec 7 '17 at 14:23
  • $\begingroup$ In contrast to the specifics of the question, this answer provided looks far from being quantitative. $\endgroup$ – NofP Dec 8 '17 at 13:58
  • $\begingroup$ 'Nobody can send a private message to someone indicating something like "Hey, how about we close this question?"' This is untrue. Any private messaging system will support this. You just can't do it inside the Stack Exchange system. You have to make a relationship with that person outside SE. $\endgroup$ – Brythan Dec 9 '17 at 1:01
  • 1
    $\begingroup$ "Regular close voters often have somewhat similar patterns." The pattern being "questions that should be closed according to those who have actual experience in the matter". $\endgroup$ – PatJ Dec 9 '17 at 17:37
  • $\begingroup$ @PatJ You're a natural comedian! Read the comments of closed questions carefully. Surprising how often knowledge and experience is absent. Minds untainted by knowledge can be remarkably confident. While this doesn't apply to all comments, it does to quite a few. $\endgroup$ – a4android Dec 12 '17 at 5:33
4
$\begingroup$

I am working on this with the data explorer.

First for some background, I started with this query and modified it to work for Worldbuilding's close criteria. I also used this query (modified) to determine how many closed questions we have had.

There were exactly 100 users who voted to close a question in 2017 that eventually became closed. Of those 100, the top 10 cast 60% of the close votes.

There were in total 1522 questions that were closed this year, so far. There are three users that have each voted to close more than 50% of the questions.

When I broke it down by month, it is not uncommon to see multiple users who have voted together to close more than 80% of the questions in a month, or a set of five users who collectively cast 50% of all close votes.

So before even doing a deeper dive analysis, I would say that, yes, there are definitely 'cliques'. However, these cliques are really just the list of people who like to close questions.

$\endgroup$
  • $\begingroup$ The first query needs a little tweak as it's only showing results for users over 10k where it should be showing over 3k. Line 22 needs updating, though the results aren't significantly different, you're missing a few users in the 100-300 close votes range. $\endgroup$ – Separatrix Dec 8 '17 at 9:49
  • $\begingroup$ @Separatrix Yeah, I linked the original query, not the one I modified. I had to do the 3k change, and remove the 'top 100' if I recall. I actually can't find the query I made now, the search features are pretty crappy. I'll link it when I find it. $\endgroup$ – kingledion Dec 8 '17 at 13:56
  • $\begingroup$ I didn't make the other changes when I ran it, only that one after it came up in chat $\endgroup$ – Separatrix Dec 8 '17 at 14:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .