github.com · LLM Evals
EleutherAI's unified harness for running hundreds of academic benchmarks against any model with consistent prompts.
lm-evaluation-harness is one of 20 LLM eval tools on TrueIQ. See all LLM eval tools or how we check facts.
// guides
Inspect AI vs DeepEval in 2026: how the two open source eval frameworks differ on who runs the model, scoring, agents, sandboxes, CI and which team each one suits.
How to use MMLU benchmark runs: pick the right variant, run it with lm-evaluation-harness or Inspect, smoke test the format, and read the score honestly.
MMLU vs GPQA explained: broad exam recall versus expert written science questions, why GPQA scores run lower, and which benchmark still means anything in 2026.
MMLU vs SWE-bench explained: what each measures, why the scores are not comparable, and which benchmark to trust for knowledge vs coding agents in 2026.
How to evaluate LLM model performance before you pick one: build a task test set, fix settings, add error bars, test prompt sensitivity, then weigh cost.
MMLU vs MMLU Pro explained: how the 4 option and 10 option benchmarks differ, why their scores aren't comparable, and which one to trust for LLMs in 2026.
// related
OpenCompassopencompass.org.cnComprehensive evaluation platform from Shanghai AI Lab supporting many models, datasets and a public leaderboard.LLM EvalsOpen source★ 7.6kOpenAI Evalsgithub.comFramework and open registry of evaluations for testing language models and the systems built on them.LLM EvalsOpen source★ 19.7klightevalgithub.comHugging Face toolkit for evaluating LLMs across many backends with reusable tasks and detailed sample level results.LLM EvalsOpen source★ 2.6ksimple-evalsgithub.comOpenAI's lightweight reference code for zero shot benchmark runs. No longer updated for new models since July 2025; it now mainly hosts HealthBench, BrowseComp and SimpleQA.LLM EvalsOpen source★ 4.7k
DeepEvaldeepeval.comPytest style framework for unit testing LLM apps with metrics for hallucination, relevance, RAG quality and more.LLM EvalsOpen source★ 19.2k
Inspect AIinspect.aisi.org.ukMIT licensed framework from the UK AI Security Institute for rigorous LLM and agent evaluations, with tool use, sandboxed tasks and 200+ prebuilt evals.LLM EvalsOpen source★ 3.1k