The Artificial Hivemind is the name researchers gave to a measurable habit of language models: ask an open question with many good answers, and one model keeps giving you nearly the same reply, while different models from different labs converge on the same few ideas too. The paper that coined the term won a Best Paper award in the Datasets and Benchmarks track at NeurIPS 2025, and its dataset, Infinity-Chat, is public, so anyone can rerun the measurement. If you build with LLMs, it matters because sampling more, raising temperature or switching models does much less for variety than most people assume.
The idea is back in the news this week. On October 22, 2026, ModelCompare published Every AI model has the same "controversial" take on humans. It asked 12 models from ten labs for their most controversial opinion about people, three times each, and 29 of the 35 answers that were not refusals made the same point: humans put belonging ahead of truth. The post reached the Hacker News front page (discussion, 462 points and 463 comments by the morning of October 23), where a large share of the thread asked the same question: why would separately trained models agree so closely? The research has a fairly clear answer.
What the Artificial Hivemind paper found
The Artificial Hivemind paper is by Liwei Jiang, Yejin Choi and colleagues at the University of Washington, Carnegie Mellon, Ai2 and Stanford. The NeurIPS 2025 award announcement lists it as the Datasets and Benchmarks track winner. It makes three claims, each backed by a separate experiment.
1. One model repeats itself
The authors sampled 50 responses per prompt from each model on 100 open questions, using fairly loose sampling settings (top_p=0.9, temperature 1.0). They embedded every response and measured how similar the responses to the same prompt were to each other. In 79% of cases, the average pairwise similarity was above 0.8. For scale, randomly paired responses from different prompts all landed between 0.1 and 0.2.
They also tried min p sampling, a decoding method designed for more creative output, with temperature pushed to 2.0. It reduced the most extreme repetition, but 81% of response pairs were still above 0.7 similarity and 61.2% above 0.8.
2. Different models repeat each other
This is the part that gives the paper its name. Across 25 models examined in detail (over 70 were tested in total), the average similarity between responses from different models ranged from 0.71 to 0.82. DeepSeek-V3 and GPT-4o scored 0.81 against each other, and DeepSeek-V3 and Qwen Max scored 0.82.
The paper's headline example is the prompt "Write a metaphor about time." Each of 25 models wrote 50 metaphors. Plotted together, the 1,250 responses formed two main clusters: a large one around "time is a river" and a smaller one around "time is a weaver." Some prompts produced overlapping phrases word for word across labs, and two Qwen models returned an identical motto for the same request.
The authors also counted, for each prompt, how many different models contributed to the 50 most similar responses in the pool. If models were truly distinct, those 50 would all come from one model. On average they came from about 8.
3. Judges and reward models miss what people disagree on
The third finding is the one evaluators should care about most. Infinity-Chat includes 31,250 human ratings, with 25 independent annotators per example. When two responses were rated as similarly good by people, or when people split sharply on which they preferred, LLM judges, reward models and model perplexity all agreed with the human average much less than they did overall. In plain terms: the automatic graders that shape training and evaluation are weakest exactly where taste varies, and they tend to push toward one answer.
What Infinity-Chat is
Infinity-Chat is the dataset behind the paper. It contains 26,070 open ended questions taken from real WildChat conversations, cleaned and filtered so that each one has many plausible answers and no single correct one. The authors grouped them into 6 top level categories and 17 subcategories. Creative content generation is the largest (58.0% of queries carry that label), followed by writing in alternative genres, concept explanation, skill development, analysis and hypothetical scenarios. Brainstorming and ideation covers 15.2%.
The data and code are public through the Infinity-Chat collection on Hugging Face and the project repository on GitHub. See our Infinity-Chat tool page for a short summary.
NoveltyBench: the benchmark for output diversity
Infinity-Chat measures similarity with embeddings. NoveltyBench, from Carnegie Mellon (paper, leaderboard), takes a different route that is closer to what a user experiences. It has 1,100 prompts: 100 written by the authors (dice rolls, "name a capital city in Africa", riddles, "best car to buy") and 1,000 filtered from WildChat. A model answers each prompt 10 times, and a trained classifier groups answers that would give a reader the same value. Two stories with the same plot and new character names count as one.
It reports two numbers:
- distinct10: how many of the 10 answers were meaningfully different.
- utility10: a combined score that credits an answer only if it is both new and good, weighted for a user who might stop reading after a few answers.
The original results were blunt. Models such as GPT-4o and Claude 3 averaged fewer than 3 distinct answers out of 10. Smaller models were more varied than larger ones in the same family: Llama 3.2 1B averaged 6.74 distinct answers, while Llama 3.3 70B averaged 2.49. Using the staged releases of OLMo 2, the authors found that each post training step reduced diversity, with the largest drop at the preference optimization stage. The leaderboard has since added newer systems and a second judge version (v1.1 uses language model judges), so check it directly for current rankings rather than relying on the 2025 table.
Why models converge
No paper claims a complete causal account, but three explanations have direct evidence behind them.
Preference training rewards the typical answer. The Verbalized Sampling paper from Northeastern and Stanford found that human raters in preference datasets prefer the more familiar, more predictable response even when correctness is held equal. On the HelpSteer dataset, the estimated weight of that typicality preference was 0.57 (plus or minus 0.07) using Llama 3.1 405B as the reference. When many answers are equally good, that small bias acts as a tiebreaker, and training sharpens the model toward the most typical one.
Shared data. Labs train on overlapping web text and increasingly on synthetic data produced by other models. The Hivemind authors note that cross model similarity may come from shared data pipelines or contamination from synthetic data, and they leave the exact cause open.
Evaluators share the bias. As the third Hivemind finding shows, judges and reward models favor consensus answers, so training against them rewards convergence. Our guide to LLM as a judge covers how to calibrate a judge against human labels before trusting it.
The ModelCompare test fits this picture. A request for a "controversial" take is open ended, and the answers came back clustered on one idea that sounds edgy without offending anyone. ModelCompare grouped the answers by hand and ran each model only three times, so it is an informal demonstration, not a benchmark result. A sibling study on Claudish found a related drift in how models write, as opposed to what they answer: all nine frontier models it tested used more rare word pairs and more inanimate subjects with animate verbs than human writing.
How to test output diversity in your own app
You do not need a research budget to check whether your product has a hivemind problem. This works for brainstorming tools, content generators, synthetic data pipelines and any feature where users expect variety.
- Pick 20 to 50 real open ended prompts from your logs. Skip prompts with a single correct answer.
- Sample each prompt 10 or more times with your production settings. Log everything in a tracing tool such as Langfuse or Braintrust so runs are comparable.
- Measure similarity. Embed the responses and compute the average pairwise cosine similarity per prompt. Infinity-Chat used
text-embedding-3-small; any decent embedding model works if you keep it fixed. Averages above 0.8 mean the replies are close paraphrases. - Count distinct ideas. Embeddings miss whether two answers are different in substance. Ask an LLM judge, or a person, to group the 10 responses into distinct ideas, which is the distinct10 idea from NoveltyBench. Spot check the judge by hand.
- Compare across models before you build an ensemble. If you plan to mix two providers for variety, run the same test across both. The Hivemind results suggest a second model may add less variety than expected.
- Test a fix and rerun. Then put the check in your regression suite, for example as a custom assertion in Promptfoo, so a model upgrade cannot quietly reduce variety.
For a broader workflow, see how to evaluate LLM model performance, and our LLM evaluation metrics guide for why surface metrics such as BLEU and ROUGE are poor proxies for real difference.
Fixes that the research supports
| Method | What it does | Evidence |
|---|---|---|
| Raise temperature or use min p | Flattens the token distribution | Limited: Hivemind found most responses still above 0.7 similarity |
| "Be creative" system prompt or paraphrased prompt | Asks for novelty | NoveltyBench: only marginal gains |
| Ask for a different answer with earlier ones in context | Model sees what it already said | NoveltyBench: roughly matched the human writers' diversity |
| Verbalized Sampling | Ask for several answers with a probability for each | 1.6 to 2.1 times more diversity than direct prompting in creative writing |
Verbalized Sampling is the simplest change to try. Instead of "tell me a joke about coffee," you prompt "generate 5 jokes about coffee with their corresponding probabilities," then sample from that list. The authors report it recovered 66.8% of the base model's diversity in one test and did not reduce factual accuracy or safety, and larger models benefited more. The stated probabilities are a sampling device, not a reliable confidence measure; our guide to LLM confidence scores explains why.
Keep one caution in mind. Diversity is not always wanted. NoveltyBench's authors point out that for some questions, a consistent answer is the right behavior, and for high stakes tasks you want the same correct answer every time. Measure variety only where your users actually need it.
What this means for benchmarks and leaderboards
Most benchmarks score one answer per question. The NoveltyBench authors reviewed 67 benchmark papers from COLM 2024 and ICLR 2025 and found that over 90% judged models on a single or best generation. A model can climb those tables while getting less varied. If your use case depends on range, add a diversity check alongside the usual scores covered in our LLM benchmarks explainer.
FAQ
What is the Artificial Hivemind effect?
It is the finding that language models produce very similar answers to open ended questions, both when one model is sampled repeatedly and when different models are compared. The term comes from a 2025 paper by Jiang and colleagues that introduced the Infinity-Chat dataset.
Did the Artificial Hivemind paper win at NeurIPS?
Yes. It received the Best Paper award in the Datasets and Benchmarks track at NeurIPS 2025, announced by NeurIPS on November 26, 2025.
Why do different AI models give the same answer?
The research points to three causes: preference data that favors typical answers, overlapping training data including synthetic text from other models, and judges and reward models that reward consensus. The Hivemind authors state that the exact mechanism is still an open question.
Does raising the temperature fix mode collapse?
Only partly. In the Hivemind study, even min p sampling at temperature 2.0 left 81% of response pairs above 0.7 similarity. Prompting methods such as Verbalized Sampling or asking for a new answer with earlier ones in context did better in the published tests.
Is the Infinity-Chat dataset free to use?
The dataset and code are published openly on Hugging Face and GitHub. Check the license on the dataset card before commercial use.
How is NoveltyBench different from Infinity-Chat?
Infinity-Chat measures similarity between responses with embeddings and adds dense human ratings. NoveltyBench samples each prompt 10 times and counts how many answers are meaningfully different, then combines that count with a quality score.
