Evals for AI applications

Your evals are only as good as their judges.

jevals gives your coding agent the tools to design an evaluation suite for your AI application, then measures every evaluator against labelled examples, so you know how often each check is right before you trust it.

Request early access Works with Claude Code and any MCP client.

Test the judges

Most eval suites score the application and never ask whether the scorer is right. An LLM judge that misses half the defects, or flags good outputs, gives you a confident number that means nothing. jevals treats every evaluator, deterministic or model-based, as something to measure.

precisionWhen it flags a defect, how often is the defect real?

target recallOf the defects it should catch, how many does it catch?

false-positive rateHow often does it fail a correct output?

How it works

Your agent reads your repository and calls jevals over MCP. You stay in the loop: it confirms failure modes with you and asks before every paid call.

Check readiness free Finds the model calls, existing evaluations, and anything that blocks measuring them.
Assess the application free Matches your patterns (RAG, tool use, pipelines, content generation, LLM judges and more) against a catalog of failure modes, and separates what you already cover from the gaps.
Design the suite paid An evaluator for each failure mode, provisional thresholds, a free gate for every pull request and a cost-capped gate for the paid model checks.
Scaffold paid A dataset layout in your repository, plus proposed controls, defect goldens and hard negatives for you to label. Nothing is overwritten.
Validate paid Checks the labelled set: every evaluator has enough controls and target defects, provenance is recorded, and the gates are sound. Every finding comes back at once.

What you end up with

Labelled goldens

Controls, paired defect goldens and hard negatives, one file each, so every label change is a reviewable diff.

Per-evaluator numbers

Precision, recall and false-positive rate for each evaluator, with reporting floors so small samples aren't over-read.

Two gates

Deterministic checks on every pull request at no cost, and model checks behind a label with a spend ceiling.

Your code, your data

Everything is written into your repository. jevals sees only what your agent sends it, such as summaries and the goldens it validates. It never reads your repository.

Terminology

The tools, the files they write and this page all use these words the same way.

The data

Dataset: One labelled collection of goldens, evaluated together. One file per golden, kept in your repository.
Suite: A named part of a dataset, grouped by why its goldens exist: broad calibration, or guarding a bug that was fixed.
Golden: One labelled example: a subject, its expected output, and why it is there. It exists before any evaluator runs.
Subject: The thing a golden asks an evaluator to judge, such as an answer, a transcript or a generated question.
Expected output: The reviewed label: is the subject defective, and should each evaluator pass it, fail it, or either.
Control: A golden that should pass. False positives are measured on controls.
Defect golden: A golden that should fail, labelled with its defect class and the evaluator meant to catch it.
Hard negative: A control made to look like a defect, so a careless evaluator would flag it.
Provenance: Who labelled a golden, on what evidence, and when. A golden without it doesn't count.

The checks

Failure mode: A way an application of a given kind tends to go wrong, such as an answer not supported by its sources. jevals keeps a catalog of them.
Defect class: A failure mode as it appears in your application, with a severity and the evaluators that should catch it.
Evaluator: Any named method that judges a subject: a deterministic check in code, or an LLM judge.
Target evaluator: The evaluator a defect golden is aimed at. Its recall is measured on exactly these goldens.

The measurement

Target recall: The share of defect goldens aimed at an evaluator that it flagged.
False-positive rate: The share of controls an evaluator flagged.
Reporting floor: The minimum number of goldens before a rate is reported, so a handful of examples isn't read as a trend.
Accepted baseline: A reviewed, committed result that later runs are compared against.
Free gate: Deterministic evaluators run on every pull request. No model calls, no cost.
Paid gate: LLM judges, run when a pull request is labelled for it, stopped at a cost ceiling.

Pricing

Working out what is worth doing is free. You pay only for the tools that build the suite, per call, from prepaid credit. No subscription. Pricing is being set during early access.

Tool	What it does
`check_readiness`	Readiness checklist with blockers and gaps	free
`assess_application`	Failure modes that apply, and which are already covered	free
`account_status`	Balance, spend today and prices	free
`design_eval_suite`	Evaluation plan: evaluators, defect classes, gates	paid
`scaffold_evals`	Dataset layout and golden proposals	paid
`validate_evals`	Every finding in the labelled set, at once	paid

Repeating an identical call within 24 hours is free, and a call that fails is not charged.
Every account has a daily spend limit.

Get started

Once you have an API key, connect your agent and give it the starter prompt.

1. Connect (Claude Code)

claude mcp add --transport http jevals https://mcp.jevals.ai/mcp \
  --header "Authorization: Bearer <api-key>"

2. Start a new session and run

/mcp__jevals__bootstrap_evals

Early access

jevals is in early access. Tell us about your application and we'll send you an API key and starting credit.

Email hello@jevals.ai