plugmyagent

DeepEval

Eval & Testing Open source

GitHub stars19k
Forks2.1k
LicenseApache-2.0
Last activityOct 19, 2026
Open issues755

Repository: confident-ai/deepeval ยท Data checked Oct 2026

About

DeepEval is the leading open-source harness for testing your LLM and agent quality over time. You install it with pip, write tests with LLM-judged metrics (faithfulness, answer relevance, and more), generate synthetic datasets, and wire it into CI so regressions get caught automatically. There is also a free cloud tier for traces if you want a dashboard.

Details

Open source. You run it yourself.

Auth: Open source

Rate limit: 100% open source; optional Confident AI cloud free tier: 2 seats, 1 project, 5 test runs/week, 1GB-month of traces

Verified: 2026-09-21

Best for: developer

Read the docs Back to catalog

Setup

  1. Run pip install deepeval.
  2. Write a test case with an input, an actual output, and expected context.
  3. Pick a metric like faithfulness or answer relevancy and run the test.
  4. Add the tests to your CI so every code change re-runs them.
  5. Optional: sign up for the free Confident AI cloud tier to view traces in a dashboard.
Sign up