github.com · LLM Evals
OpenAI's lightweight reference code for zero shot benchmark runs. No longer updated for new models since July 2025; it now mainly hosts HealthBench, BrowseComp and SimpleQA.
simple-evals is one of 20 LLM eval tools on TrueIQ. See all LLM eval tools or how we check facts.
// guides
The best LLM evaluation frameworks in 2026: Promptfoo, DeepEval, Ragas, Inspect and TruLens for apps, plus lm-evaluation-harness and HELM for model benchmarks.
// related
OpenAI Evalsgithub.comFramework and open registry of evaluations for testing language models and the systems built on them.LLM EvalsOpen source★ 19.7k
OpenCompassopencompass.org.cnComprehensive evaluation platform from Shanghai AI Lab supporting many models, datasets and a public leaderboard.LLM EvalsOpen source★ 7.6klm-evaluation-harnessgithub.comEleutherAI's unified harness for running hundreds of academic benchmarks against any model with consistent prompts.LLM EvalsOpen source★ 14.3klightevalgithub.comHugging Face toolkit for evaluating LLMs across many backends with reusable tasks and detailed sample level results.LLM EvalsOpen source★ 2.6k
DeepEvaldeepeval.comPytest style framework for unit testing LLM apps with metrics for hallucination, relevance, RAG quality and more.LLM EvalsOpen source★ 19.2k
Inspect AIinspect.aisi.org.ukMIT licensed framework from the UK AI Security Institute for rigorous LLM and agent evaluations, with tool use, sandboxed tasks and 200+ prebuilt evals.LLM EvalsOpen source★ 3.1k