The week in brief
Behind it, Promptfoo (+198) and Opik (+134) followed.
Three new tools joined the directory this week: SWE-Bench Pro, OpenAI Content Verification and Humanity's Last Exam (HLE).
On Hacker News, “Benchmark in Milliseconds” drew the most attention in this space (148 points, 53 comments). Among listed tools, Artificial Analysis's site was linked from 84 Hacker News stories over the past 90 days, more than any other tool here.
In research, “EmbodiedCLBench: Evaluating Continual Learning for Self Evolving Embodied Agents” collected the most upvotes on Hugging Face Daily Papers this week (56). Among new GitHub repositories created in the past 7 days, pipelinelabai/paperbench-expanded leads with 51 stars.
Biggest GitHub star gainers
GitHub star counts for the 35 open source tools we track, compared between Oct 5, 2026 and Oct 12, 2026 (155 hours apart).
| Tool | Stars before | Stars after | Gain | Change |
|---|---|---|---|---|
| Langfuse | 35,410 | 35,688 | +278 | 0.79% |
| Promptfoo | 25,730 | 25,928 | +198 | 0.77% |
| Opik | 22,391 | 22,525 | +134 | 0.60% |
| DeepEval | 18,649 | 18,762 | +113 | 0.61% |
| Aider Leaderboards | 49,384 | 49,489 | +105 | 0.21% |
| Arize Phoenix | 11,716 | 11,805 | +89 | 0.76% |
| Ragas | 15,937 | 16,001 | +64 | 0.40% |
| lm-evaluation-harness | 14,134 | 14,192 | +58 | 0.41% |
New listings this week
| Tool | Category | Listed |
|---|---|---|
| SWE-Bench Pro | Benchmarks | Oct 10, 2026 |
| OpenAI Content Verification | Integrity & Detection | Oct 08, 2026 |
| Humanity's Last Exam (HLE) | Benchmarks | Oct 07, 2026 |
Most discussed on Hacker News
The highest scoring Hacker News stories in this space during the week.
Listed tools with the most Hacker News links
Hacker News stories linking to each tool's own website over the past 90 days.
Notable new papers
The most upvoted papers on Hugging Face Daily Papers and the newest relevant arXiv preprints from this week.
- EmbodiedCLBench: Evaluating Continual Learning for Self Evolving Embodied Agents
- Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long Running Game Agent Competition
- Pumpire: Unified Benchmark for Metric Distance Estimation
- Towards Physically Grounded Articulated 3D Assets: A Multi Agent Pipeline and Benchmark
- GraphDecisionBench: Benchmarking System One Models on Graph Tasks
- QTestBench: A Test Framework for Evaluating LLM Generated Quantum Code
- FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems
- Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models
- SeoulMMOD: A Large Scale Multimodal Origin-Destination Flow Benchmark
- Retractions Revisited: A Replication, Extension, and Mechanistic Evaluation of Linguistic Obfuscation Effects in Fraudulent Science
New GitHub repos in LLM evaluation
Repositories created in the 7 days before Oct 12, 2026, ranked by stars.
| Repository | Stars | Language | Description |
|---|---|---|---|
| pipelinelabai/paperbench-expanded | 51 | Python | PaperBench High Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain native software and graded by clean replay. 12 public tasks. |
| talhaanwarch/awesome-llmops-platforms | 33 | Python | Curated list of open source LLMOps tools: tracing, evals, AI gateways, guardrails and prompt management. Sorted by stars, refreshed weekly. |
| rasswanth-lyzr/ship-ai-projects | 6 | Commented SHIP? Full briefs for 3 pile two AI engineering projects: LLM Ops, Agent Evals, Auto Routing. What to build, what to measure, what to write in the README. | |
| Mahalanobis-Systems/aot-evals | 5 | Python | AOT Evals makes agent evals easy. Create golden records, safe testing environments, and repeatable optimization loops to improve cost, quality, and speed of your agent. Get started in 5 minutes. |
| deyna256/convy | 4 | Python | Conversation tests for AI agents: a simulated user talks to your agent through its API, a judge checks every claim, and convy reports pass rate, tokens, time and what changed between builds. |
| ashishpatel26/harness-eval | 3 | Python | Did this change to the coding agent harness make the code better or worse? Paired, repeated, isolated evaluation with a verdict you can trust. |
| mahingbun-dev/ai-evo | 3 | Python | Self hosted AI evaluation platform for LLMs & Agents: 9 eval types with AI judges, datasets, pre eval scripts, analytics & A/B testing. FastAPI + Vue 3. |
| zlab-princeton/sweeper-bench | 2 | Python | SWEeper-Bench: Can agents discover bugs in interactive software? 200 real bugs in 200 open source web apps; agents get only the code and an open ended prompt, and an agentic verifier checks each fix i |
How we compile this report
Every figure comes from a public source: star counts from the GitHub API, story scores and links from Hacker News (via Algolia), papers from arXiv and Hugging Face Daily Papers, and new repositories from GitHub search. Hacker News link counts include stories whose link points to the tool's own domain. Listing dates are the day a tool joined TrueIQ. Numbers are shown as recorded; if a figure is unavailable, it is left out rather than estimated.