inspect.aisi.org.uk · LLM Evals
MIT licensed framework from the UK AI Security Institute for rigorous LLM and agent evaluations, with tool use, sandboxed tasks and 200+ prebuilt evals.
Inspect AI is one of 20 LLM eval tools on TrueIQ. See all LLM eval tools or how we check facts.
// guides
Inspect AI vs DeepEval in 2026: how the two open source eval frameworks differ on who runs the model, scoring, agents, sandboxes, CI and which team each one suits.
How to use MMLU benchmark runs: pick the right variant, run it with lm-evaluation-harness or Inspect, smoke test the format, and read the score honestly.
How to evaluate LLM model performance before you pick one: build a task test set, fix settings, add error bars, test prompt sensitivity, then weigh cost.
OpenAI Evals vs DeepEval in 2026: platform shutdown timeline, DeepEval's pytest CI metrics, multiprovider support, and Promptfoo as the migration path.
The best LLM evaluation frameworks in 2026: Promptfoo, DeepEval, Ragas, Inspect and TruLens for apps, plus lm-evaluation-harness and HELM for model benchmarks.
How to evaluate LLM applications step by step: error analysis, code checks, calibrated LLM judges, CI gates and production sampling, with tools for each stage.
// related
PyRITmicrosoft.github.ioMicrosoft's open source (MIT) Python Risk Identification Tool for red teaming generative AI, with composable targets, converters, scorers and multiturn attacks.LLM EvalsOpen source★ 4.7k
DeepTeamtrydeepteam.comOpen source (Apache 2.0) LLM red teaming framework from Confident AI, built on DeepEval, covering 50+ vulnerability types with OWASP, NIST and MITRE presets.LLM EvalsOpen source★ 3.1k
garakgarak.aiNVIDIA's open source (Apache 2.0) LLM vulnerability scanner that probes models and endpoints for jailbreaks, prompt injection, data leakage and toxic output.LLM EvalsOpen source★ 9.7k
mlprintsgithub.comOpen source AGPL-3.0 framework from Sentient for implementing, attacking and verifying LLM fingerprinting methods in a shared evaluation environment.LLM EvalsOpen source★ 3
Galileogalileo.aiEvaluation, observability and guardrail platform for AI agents and LLM apps. Cisco completed its acquisition in May 2026 and is folding it into Splunk Observability.LLM EvalsFreemium
DeepEvaldeepeval.comPytest style framework for unit testing LLM apps with metrics for hallucination, relevance, RAG quality and more.LLM EvalsOpen source★ 19.2k