evalharness is pytest for LLM outputs — golden sets, LLM-as-judge scoring, and a regression gate that fails the build when quality drops. Pick one of 10 AI features, edit its prompt, and watch your prompt quality move the score — live.
Versioned datasets of inputs, expected answers, and grading rubrics — your ground truth, in YAML or JSONL.
A strong model grades open-ended output against a rubric and returns a schema-validated verdict — not brittle string matching.
Exact, regex, contains, local semantic embedding, and LLM-judge — composed per suite, each measuring one thing.
Compare any run to a saved baseline. Catch cases that flip pass→fail and aggregate drops beyond your tolerance.
One command exits non-zero when quality drops. A prompt or model change that regresses turns the build red.
A content-addressed response cache makes re-runs free and gate decisions deterministic despite model randomness.
This is the opposite: a reusable framework with a pinned, cached, reproducible pipeline; composable scorers including a structured LLM judge; an explicit regression algorithm; and a gate that actually fails the build.
Every change is graded against your baseline. If quality holds, the gate is green and you ship. If it slips, the gate goes red and tells you exactly what broke — down to the case.
Want to trigger both yourself? Open the dashboard and use the one-click guided demo.