Is there hard data on users' cliques for close votes?

Question

I was reading a previous question on data on close votes, and a related question came to my mind.

The question is as follows: is it possible to reject the hypothesis that there exists well-defined cliques of users that have a higher than random frequency in co-voting to close questions?

As a noob in terms of community dynamics, my expectation would be that the specific sample of users that vote to close any given question should be a random sample of the the population of users that in general cast a vote to close questions.

If that is not the case, I am of the opinion that there could be other factors at play, some of which should be considered and addressed as a community. For instance, there could be cascade effects, such that a user vote to close prompts a fellow user (let's say a friend) to vote to close, regardless of the merit of the question per se; or, it could be topic driven, i.e. some groups of users do not like (otherwise reasonable) questions on certain topics, hence the co-voting patterns; or, it could be that there is a group strategy (explicit or just emergent) to assert dominance and control within the larger group of the community. You name it.

I would tag this question as hard-statistics, in case.

Answer 1

Data

I analyzed all the data for 2017. I got the list of question-closers for 1719 questions that have been closed for any reason but not deleted (deleted posts usually have no upvotes or answers after 30 days). This also counts questions that have since been reopened. I then took every subset of these closers and counted them. Here are the results

Groups of 1

The most closing user voted on 907 close questions, or 53% of the total. There are 11 users who voted on 10% or more of the close questions and those 11 users placed 2.9 votes per closed question. Or, in other words, they provided 59 % of all close votes.

Groups of 2

The highest grouping of 2 voted to close 569 questions together, or 33% of all questions. There were 10 pairs of users from 6 individual users who voted on 10% or more of closed questions together, and they placed 1.6 votes per question, or 33 % of the total close votes.

Groups of 3 or more

The highest grouping of 3 users voted to close 187 questions together. The highest group of 4 was 48 questions, and the highest group of 5, only 7 questions.

Gini Index

The Gini index of close voters is 0.807. Gini is a measure of inequality, used for incomes mostly. I don't have any information to put this number in context, but I am interested in doing so in the future, by comparison over time, or with other sites.

Pearsons $\chi^2$ test for independence

Forming the the 11 highest users into a diagonal contingency table, we can use Pearsons test for independence. Using the null hypothesis of independence, I built a contingency table with 50 degrees of freedom. That table had a Pearson's $\chi^2$ score of 5492, which is very high. Since $p = 0$ for this test, we can (resoundingly) reject the null hypothesis.

A cursory look at the data shows us why.

Some users close a lot of questions

Especially recently, there are two users who have closed 'most' of the questions. Since August 1, there have been 718 questions closed; these two users closed 75% and 73% respectively.

These two users have voted together 399 times, or 76% and 74% of those users total close votes, respectively. Together, they have voted to close 56% of all questions.

There is a third user who frequently votes along side these two. This third user has 207 close votes, of which 57% have been with both of the two high close voters, and 93% have been with one or the other.

This appears to be evidence of some 'collusion.' However, when comparing the actual number of times users have closed questions against expected, even in these recent months when many questions have been closed, they are not operating together more than expected. Here are actual and expected numbers for users A, B, and C from above:

Users   Expected Closes   Actual Closes
A + B              393              399
A + C              157              159
B + C              152              152
A, B, C            113              119

In fact, the expected values are surprisingly accurate.

If we redo the $\chi^2$ test for this time period, we see that the score has dropped all the way down to 138. This is still low enough to reject the null hypothesis ($p=0.008$), but it is much closer than before.

Conclusion

It does look like certain groups are closing questions together a lot as a clique. However, this behavior is expected due to how frequently members of that group are closing questions in general. Despite a failure of independence test for the entire year, when broken down into smaller samples, the test looks a lot better.

Therefore, we can reject the null hypothesis that close voters are acting independently, but we can demonstrate some evidence that time is the factor causing this result. Overall, I conclude that there is insufficient evidence of voters demonstrating any of the co-voting patterns that you mention in the question.

Answer 2

I don't have any data at hand, but here's my feeling based on what I've seen on the site and a few things to keep in mind for this topic.

You can't see who voted to close before the closing is complete, so long as people don't explicitly comment.

If I see a question in the review queue and no comments indicating any problems I often leave a Reminder to Close-Voters to prompt others to state what they think is wrong with the question. Other people do this regularly, too, if there hasn't been any comment. So the second or third person to get a close vote in the review queue can often see at least one person who said something that may be wrong with a question.

But still: you normally don't know who voted to close until the question is put on hold. Just like you can't normally see who voted to reopen until the question is reopened.

You could go to your review history after casting your vote and see who voted for what, but that's already after the fact.

This makes it hard for cliques to know when their voting-buddy voted for something.

There is no private messaging integrated into the SE network.

Nobody can send a private message to someone indicating something like "Hey, how about we close this question?" inside of the normal network.

It would require the participants to get acquainted outside of the network, which increases the amount of work slightly that would be required, just for this kind of action.

The closest you can get inside of the normal SE network is the chat, which everybody can access to write who has at least 20 reputation. And everybody can read it, no matter the reputation. This is a very low mark and there are no other requirements to be part of this "clique". In fact we would love if more people regularly visited the chat to talk about their ideas/projects/... I've seen it sometimes (maybe once a month or every two months?) that someone posts a blatantly off-topic question there to start a discussion with the regulars who are online at the moment. Or about a question that needs reopening, because the author edited his question. Or about asking people to help with editing to give a new user some hints. A lot of the regulars in the chat have enough reputation to vote for close/reopen/edit/..., which makes this the ideal place to talk about pro and con of putting something on hold.

There was even a special chat room for talking about reopening.

Bottom line: the chat may be somewhat active at times, but there is not even enough attention to maintain a chatroom for this topic. The special room was frozen and it's very rare that a question pops up in the chat for VTC/VTO. And the discussions tend to be quite diverse. The people there are not thinking "Ah, I know that username, so the question he mentioned must be closed immediately." - we openly talk about why a question should be put on hold or reopened if the case arises. Still, you could call it a "clique" if you wanted to. It's the closest thing you can get to a "clique" on this site anyway as the amount of regular chat users is quite limited, although everyone with a very little amount of 20 reputation can be part of this very open "clique".

There are Meta discussions.

More often than in the chat someone posts something about closing/reopening on Meta. This leads to the regular Meta users seeing the question and acting on them, as a lot of them also have higher rep. Again, this could be seen as a "clique", even if it's open to everyone. It's commonly referred to as the Meta Effect. But these Meta discussions are still not something that leads to people thinking "Ah, yes, gotta close". They are open discussions where different people offer different views.

People can post on Meta with 5 reputation. Again, reading is possible even for people that are not logged in to the site. It's open for everyone. And 5 reputation is a single answer or question upvote. It's a tenth of the reputation you need to be able to comment everywhere. This could be seen as a "clique", but just like the chat, it's a very open and welcoming "clique". More activity on Meta would also be a great thing. There can't be too many people active in the chat / on Meta, so getting more opinions about how closing/reopening/... should work on the site would be great and people listen to arguments - not to the username that posted something.

Regular close voters often have somewhat similar patterns.

I know that there are a handful of people that very often vote similar to how I am voting. Seeing their username in the comments does in fact draw my personal attention because I know how they comment and realized through experience that they are very good at filtering the noise or summarizing problems/possible solutions. Data would surely suggest that we are a "clique" - mainly because they just vote similar most of the time.

Just because people vote similar doesn't mean they stop thinking the moment one of them posts something. For example those people I have in mind have a similar opinion most of the time - just not when it comes to questions tagged magic. There are surely similarities in the voting pattern, but also bigger differences.

A question that is put on hold without any real reason can easily be reopened.

Posting on Meta and asking for the reasons for a question that was put on hold will draw in people that look at the question and you will surely find people that see the merit. If there are enough the question can be reopened. And as you can only VTC once on every question there won't be a war. You may see patterns of people who don't like magic questions for example voting to clsoe these questions and people who like them and vote to reopen - in the end the whole active community decides.

People show dramatically different levels of activity when it comes to reviewing.

Some people review a lot. Some people don't like reviewing at all.

You will surely find that a few people VTC very often, especially compared with the average user. You could say that they are a "clique". It would probably look like it. But someone has to do the reviewing if we want high quality questions and answers on the site. If you feel there are too few reopen votes for example the best thing to do is to lead with a good example by raising the topic on Meta/in chat regularly and trying to focus on the reopen review queue. But just like you might want to focus on the reopen queue you will find people who focus on the closing queue. Because that's also an important part of the community.

Conclusion

I don't think there are "cliques" of people on this site. Yes, there are some people who are more strict/lenient when it comes to certain topics and there are groups that you will find more often as reviewers, such as regular Meta users and regular chat users, but it's not like they are always voting for the same thing or stop to think once they see someone vote. And there are no groups who regularly invite others to close a bunch of questions they don't like. This would probably have the adverse effect quite often - if you can't say why you want to close something I for example tend to give the querent the benefit of doubt.

But I am sure that any data would reveal that some people are very often co-voting. I can think of a few people that I would expect to see with a similar voting pattern to mine because we have a similar take on a lot of topics. Participating in a lot of discussions about these topics, because we always had them and will always have them on this site, leads to that, which is also a good sign that there are certain rules that people can follow to know whether something is on-topic or off-topic.

Answer 3

I am working on this with the data explorer.

First for some background, I started with this query and modified it to work for Worldbuilding's close criteria. I also used this query (modified) to determine how many closed questions we have had.

There were exactly 100 users who voted to close a question in 2017 that eventually became closed. Of those 100, the top 10 cast 60% of the close votes.

There were in total 1522 questions that were closed this year, so far. There are three users that have each voted to close more than 50% of the questions.

When I broke it down by month, it is not uncommon to see multiple users who have voted together to close more than 80% of the questions in a month, or a set of five users who collectively cast 50% of all close votes.

So before even doing a deeper dive analysis, I would say that, yes, there are definitely 'cliques'. However, these cliques are really just the list of people who like to close questions.