promptfoo.dev · LLM Evals
Open source CLI for testing prompts, models and RAG apps with assertions, side by side comparisons and automated red teaming. Part of OpenAI since 2026 and still MIT licensed.
Promptfoo is one of 20 LLM eval tools on TrueIQ. See all LLM eval tools or how we check facts.
// guides
Inspect AI vs DeepEval in 2026: how the two open source eval frameworks differ on who runs the model, scoring, agents, sandboxes, CI and which team each one suits.
The Artificial Hivemind paper explained: why AI models give the same answers, what Infinity-Chat and NoveltyBench measure, and how to test output diversity yourself.
OpenAI Evals vs DeepEval in 2026: platform shutdown timeline, DeepEval's pytest CI metrics, multiprovider support, and Promptfoo as the migration path.
What a prompt injection attack is, direct and indirect examples, why there is no complete fix, and how to test your LLM app with Promptfoo, garak and PyRIT.
The best LLM evaluation frameworks in 2026: Promptfoo, DeepEval, Ragas, Inspect and TruLens for apps, plus lm-evaluation-harness and HELM for model benchmarks.
The best LLM red teaming tools in 2026 compared: Promptfoo, garak, PyRIT, DeepTeam and Giskard, with licenses, first commands and OWASP LLM Top 10 coverage.
// related
garakgarak.aiNVIDIA's open source (Apache 2.0) LLM vulnerability scanner that probes models and endpoints for jailbreaks, prompt injection, data leakage and toxic output.LLM EvalsOpen source★ 9.7k
Giskardgiskard.aiOpen source Python library for evaluating and red teaming AI agents and RAG apps. Version 3 is a modular rewrite for multiturn testing; v2 is no longer maintained.LLM EvalsOpen source★ 5.9k
PyRITmicrosoft.github.ioMicrosoft's open source (MIT) Python Risk Identification Tool for red teaming generative AI, with composable targets, converters, scorers and multiturn attacks.LLM EvalsOpen source★ 4.7k
DeepTeamtrydeepteam.comOpen source (Apache 2.0) LLM red teaming framework from Confident AI, built on DeepEval, covering 50+ vulnerability types with OWASP, NIST and MITRE presets.LLM EvalsOpen source★ 3.1k
mlprintsgithub.comOpen source AGPL-3.0 framework from Sentient for implementing, attacking and verifying LLM fingerprinting methods in a shared evaluation environment.LLM EvalsOpen source★ 3
DeepEvaldeepeval.comPytest style framework for unit testing LLM apps with metrics for hallucination, relevance, RAG quality and more.LLM EvalsOpen source★ 19.2k