Promptfoo is the best LLM evaluation tool: it is free under the MIT license, scores outputs with metrics you define, compares models side by side and runs checks in CI/CD. DeepEval is #2 with the widest ready-made metric list and a cloud plan from $0. Evidently AI is #3, with 100+ built-in metrics and pass/fail Test Suites.
The open-source library is free under Apache-2.0; the README says Evidently Cloud has a free tier, and no Cloud price is published on the pages we read
Teams that also need drift and predictive ML checks next to their LLM evals
Free and open source under the MIT license; no paid plan is listed on the pages we read
Researchers benchmarking a model on standard academic tasks with reproducible prompts
Pick by what you want
Best overall
Promptfoo
Widest ready-made metric list
DeepEval
Best for multi-turn agents
Giskard
Best hosted workflow
Braintrust
Best for RAG metrics
Ragas
Best for model benchmarks
lm-evaluation-harness
The ranking
1
Promptfoo
Answrs VerdictScoring, model comparison and CI/CD checks in one free MIT tool.
Price: The Community plan is free forever; Enterprise and On-Premise are custom pricedLicense: MIT license; GitHub shows 26,942 starsWhat it does: An open-source CLI and library for evaluating and red-teaming LLM apps; you score outputs automatically by defining metricsWorkflow: The README lists automated evaluations, side-by-side model comparison across OpenAI, Anthropic, Azure, Bedrock and Ollama, and automated checks in CI/CDOwnership: The README says Promptfoo is now part of OpenAI and remains open source and MIT licensed
Promptfoo is free under the MIT license, and its Community plan is free forever; Enterprise and On-Premise are custom priced. It is an open-source CLI and library for evaluating and red-teaming LLM apps, scoring outputs automatically with metrics you define. Its README lists side-by-side comparison across OpenAI, Anthropic, Azure, Bedrock and Ollama and automated checks in CI/CD, and says Promptfoo is now part of OpenAI.
Go ifDevelopers who want a free CLI that runs prompt and model comparisons in CI/CD
Skip ifYou want a hosted team dashboard with a published paid price
Why #1 over #2: It documents scoring, CI/CD and agent testing and is MIT licensed, while DeepEval puts its hosted plans behind a paid ladder.
2
DeepEval
Answrs VerdictThe widest ready-made metric list here, with a published cloud price ladder.
Price: Confident AI cloud is $0 on Free, $200 a month on Starter and $2,000 a month on Team; Enterprise is customLicense: The DeepEval framework is Apache-2.0 licensed; GitHub shows 19,357 starsFree cloud limits: 2 user seats, 1 project, 5 test runs per week and 1 GB-month of trace spansMetrics: The README lists ready-to-use metrics for agents, voice agents, multi-turn chats, RAG, MCP and images, including G-Eval, Task Completion, Tool Correctness, Faithfulness and Contextual PrecisionTesting: The README calls it similar to Pytest but specialized for unit testing LLM apps, and its docs list Unit Testing in CI/CD
DeepEval is Apache-2.0 licensed, and its Confident AI cloud is $0 on Free, $200 a month on Starter and $2,000 a month on Team. Free allows 2 user seats, 1 project and 5 test runs per week. Its README lists metrics for agents, voice agents, multi-turn chats, RAG, MCP and images, and calls it similar to Pytest but specialized for unit testing LLM apps.
Go ifPython teams that want pytest-style LLM tests and a metric for agents, RAG and chats
Skip ifYou want the hosted platform for free beyond five test runs a week
Why #2 over #3: It has a wider ready-made metric list than Evidently AI, which also covers classic ML metrics.
3
Evidently AI
Answrs VerdictLLM judges plus classic ML metrics with pass/fail Test Suites for CI.
Price: The open-source library is free under Apache-2.0; the README says Evidently Cloud has a free tier, and no Cloud price is published on the pages we readLicense: Apache-2.0; GitHub shows 8,047 starsMetrics: 100+ built-in metrics, from data drift detection to LLM judgesTest suites: Any Report can become a Test Suite with pass/fail conditions, which the README recommends for regression testing and CI/CD checksScope: The site says it evaluates, tests and monitors LLMs, RAG applications, AI agents and ML models in a single framework
Evidently AI is Apache-2.0 with 100+ built-in metrics, from data drift detection to LLM judges. Any Report can become a Test Suite with pass/fail conditions, which the README recommends for regression testing and CI/CD checks. The README says Evidently Cloud has a free tier, but no Cloud price is published on the pages we read.
Go ifTeams that also need drift and predictive ML checks next to their LLM evals
Skip ifYou want a published price for the hosted Cloud plan
Why #3 over #4: It documents CI/CD-style Test Suites, while Giskard focuses on agent scenarios.
Price: The open-source library is free under Apache-2.0; Giskard Hub is an enterprise platform and no price is published on the pages we readLicense: Apache-2.0; GitHub shows 5,947 starsBuilt-in checks: Built-in evals include string matching, comparisons, regex, semantic similarity and LLM-as-judge checks such as Groundedness, Conformity and LLMJudgeAgents: The README says to use it to catch regressions and to evaluate multi-turn agents, testing full conversations rather than single exchangesVersion 3: Giskard v3 is a rewrite for multi-turn testing of AI agents and requires Python 3.12+; the README says v2 remains available but is no longer actively maintained
Giskard is free under Apache-2.0, with built-in checks for string matching, regex, semantic similarity and LLM-as-judge such as Groundedness. The README says to use it to catch regressions and to evaluate multi-turn agents. Giskard v3 requires Python 3.12+, and v2 is no longer actively maintained; Giskard Hub is an enterprise platform with no published price.
Go ifTeams testing multi-turn agents who want regression checks plus a vulnerability scan
Skip ifYou are on Python older than 3.12 or still rely on v2
Why #4 over #5: It documents multi-turn agent testing and regression checks, while Braintrust is a hosted product with no open-source license listed.
5
Braintrust
Answrs VerdictA full hosted eval cycle with a published $249 Pro plan.
Price: Starter is $0 a month; Pro is $249 a month; Enterprise is customStarter limits: 1 GB processed data, 10k scores and 14-day retention, with unlimited users, projects, datasets, playgrounds and experimentsPro limits: 5 GB processed data, 50k scores and 30-day retentionEvaluators: An evaluation has data, a task and evaluators; evaluators are scorers and classifiers built with autoevals, LLM-as-a-judge or custom codeWorkflow: The docs describe playgrounds, experiments, automating evals in CI/CD on every pull request, and online scoring of live traffic
Braintrust Starter is $0 a month with 1 GB processed data, 10k scores and 14-day retention; Pro is $249 a month with 5 GB, 50k scores and 30-day retention; Enterprise is custom. Its docs describe playgrounds, experiments, CI/CD evals on every pull request and online scoring of live traffic.
Go ifTeams that want a hosted workflow from playground to CI to production scoring
Skip ifYou need self-hosting without an Enterprise contract
Why #5 over #6: It documents the full cycle from playground to CI, while LangSmith leans on its own tracing platform.
6
LangSmith
Answrs VerdictOffline and online evaluation with per-seat pricing from $39.
Price: Developer is $0 per seat a month with 1 seat; Plus is $39 per seat a month with unlimited seats; Enterprise is customBase traces: Up to 5k base traces a month on Developer and up to 10k on Plus, then pay as you goEvaluation: Scores agent outputs against datasets and criteria so you can benchmark versions, catch regressions and track quality over timeOffline and online: Offline evaluation runs on curated datasets during development; online evaluation runs on real user interactions in production
LangSmith Developer is $0 per seat a month with 1 seat and Plus is $39 per seat a month with unlimited seats; Enterprise is custom. Its evaluation docs cover offline evaluation on curated datasets and online evaluation of real user interactions, to benchmark versions and catch regressions.
Go ifTeams already building agents with LangChain who want datasets, evaluators and production scoring
Skip ifYou want a flat price instead of per-seat billing
Why #6 over #7: It covers all three needs, while Ragas does not document CI/CD or regression testing.
7
Ragas
Answrs VerdictDeep RAG and agent metric lists, but no CI workflow documented.
Price: Free open-source library under the Apache-2.0 license, installed with pip install ragasLicense: Apache-2.0; GitHub shows 16,258 stars; the latest push to the repository was 24 February 2026RAG metrics: The metrics list groups Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Response Relevancy and Faithfulness under Retrieval Augmented GenerationAgent metrics: Topic Adherence, Tool Call Accuracy, Tool Call F1 and Agent Goal Accuracy are listed for agents or tool useTest data: Includes production-aligned test set generation for when you do not have a test dataset ready
Ragas is a free Apache-2.0 library installed with pip install ragas. Its metrics list covers Context Precision, Context Recall, Noise Sensitivity, Response Relevancy and Faithfulness for RAG, plus Topic Adherence, Tool Call Accuracy and Agent Goal Accuracy for agents. The latest push to the repository was 24 February 2026.
Go ifTeams evaluating RAG pipelines or tool-using agents who want a metrics library
Skip ifYou need CI/CD or regression testing documented, which the pages we read do not describe
Why #7 over #8: It lists RAG and agent metrics, while TruLens documents LLM judges but not a metric catalog this long.
8
TruLens
Answrs VerdictFree explainable LLM judges, though its benchmark figures are self-reported.
Price: Free and open source under the MIT license; no paid plan is listed on the pages we readLicense: MIT; GitHub shows 3,629 starsScoring: Instrument any app with a decorator and score every step with LLM judges that explain themselves; evaluations run as traces land or over a datasetConversations: The site announces support for tracking and evaluating complete multi-turn conversationsSelf-reported benchmark: TruLens reports 95% of agent errors caught with Agent GPA on TRAIL and GAIA, 267 of 281 human-annotated errors, against 55% for a baseline trace judge; this is TruLens reporting on itself
TruLens is MIT licensed with no paid plan listed on the pages we read. You instrument an app with a decorator and score every step with LLM judges that explain themselves, and its site announces multi-turn conversation evaluation. It reports 95% of agent errors caught with Agent GPA, 267 of 281 annotated errors; that figure is TruLens reporting on itself.
Go ifTeams that want free LLM judges that explain their scores, with OpenTelemetry tracing
Skip ifYou need CI/CD or regression testing documented, which the pages we read do not describe
Why #8 over #9: It documents agent and multi-turn scoring, while Inspect AI centers on pre-built evaluations.
9
Inspect AI
Answrs VerdictOver 200 pre-built evaluations and sandboxed agents from a government-backed lab.
Price: Free and open source under the MIT license; no paid plan is listed on the pages we readLicense: MIT; GitHub shows 3,201 starsPre-built evals: A collection of over 200 pre-built evaluations ready to run on any modelAgents: Supports agent evaluations, including running external agents such as Claude Code, Codex CLI and Gemini CLISandboxing: Runs untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant and other systems via an extension API
Inspect AI is MIT licensed and made by the UK AI Security Institute and Meridian Labs. It ships over 200 pre-built evaluations, supports agent evaluations including external agents such as Claude Code, Codex CLI and Gemini CLI, and sandboxes untrusted model code in Docker, Kubernetes, Modal, Proxmox and Vagrant.
Go ifSafety and capability researchers who want 200+ ready evaluations and sandboxed agent runs
Skip ifYou need CI/CD or a hosted dashboard, which the pages we read do not describe
Why #9 over #10: It documents agent evaluation, while lm-evaluation-harness only benchmarks models.
10
lm-evaluation-harness
Answrs VerdictThe standard benchmark runner for models, not for testing apps or agents.
Price: Free and open source under the MIT license; no paid plan is listed on the pages we readLicense: MIT; GitHub shows 14,424 starsBenchmarks: Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variantsBackends: Supports transformers, GPT-NeoX, Megatron-DeepSpeed, vLLM, SGLang, OpenAI and TextSynth APIs, and PEFT adapters such as LoRALeaderboard tasks: The README says Open LLM Leaderboard tasks were added under a leaderboard task group
lm-evaluation-harness is MIT licensed from EleutherAI, with over 60 standard academic benchmarks and hundreds of subtasks and variants. It supports transformers, GPT-NeoX, Megatron-DeepSpeed, vLLM, SGLang, the OpenAI and TextSynth APIs and PEFT adapters, and the README says Open LLM Leaderboard tasks were added.
Go ifResearchers benchmarking a model on standard academic tasks with reproducible prompts
Skip ifYou want to test your own app, prompts or agents rather than score a model
How we ranked these
We chose eval-focused tools and kept no more than three picks shared with any one other live answrsnow page, so MLflow, Arize Phoenix and Langfuse are covered elsewhere on the site. Tools were ranked by how many of three needs each documents on its own pages: built-in scoring, CI/CD or regression testing, and agent or multi-turn evaluation. Ties go to the tool with more GitHub stars; hosted-only tools follow open-source ties in alphabetical order. Prices are the published list prices, and where a price is not published we say so. Benchmark claims are labelled as the maker’s own. OpenAI Evals and Humanloop are not included because OpenAI is shutting its Evals platform down and Humanloop has sunset its platform. Facts are taken from each maker's own published information; check the official site before you buy.
Questions people ask
What is the best LLM evaluation tool?
Promptfoo ranks first: it is free under the MIT license, scores outputs with metrics you define, compares models side by side and runs checks in CI/CD. DeepEval and Evidently AI also document all three needs we checked.
Which LLM evaluation tools are free?
Promptfoo and TruLens are MIT licensed, and Inspect AI and lm-evaluation-harness are MIT licensed too. DeepEval, Evidently AI, Giskard and Ragas are Apache-2.0. Hosted plans: Braintrust Starter and LangSmith Developer are $0.
What do the paid hosted plans cost?
LangSmith Plus is $39 per seat a month, Braintrust Pro is $249 a month, DeepEval maker Confident AI lists $200 a month for Starter and $2,000 a month for Team. Evidently AI and Giskard do not publish hosted prices on the pages we read.
What is the difference between benchmarking a model and testing an app?
lm-evaluation-harness runs standard academic benchmarks on a model. Promptfoo, DeepEval, Braintrust and LangSmith test your own prompts, RAG pipelines or agents against your own data.
GitHub stars of the open-source projects (checked 7 November 2026) at a glance
Inspect AI3,201 stars
TruLens3,629 stars
Giskard5,947 stars
Evidently AI8,047 stars
lm-evaluation-harness14,424 stars
Ragas16,258 stars
DeepEval19,357 stars
Promptfoo26,942 stars
GitHub stars of the open-source projects (checked 7 November 2026), as published by each maker. Picks without a published figure are left off.
