Run a Suite
Enqueue every case under the suite as a fresh eval_runs row. Returns immediately
with a synthetic suite_run_id that groups the spawned runs. Workers pick up the
runs and process up to suite.parallelism at a time.
Required scope agent_eval:run. The runs are billed to the company’s credit balance
as they finish — see /billing/consumption?product=eval_run for an after-the-fact view.
Suites are always processed asynchronously — poll GET /agent-eval/suites/{id}/runs/{suite_run_id}
until in_flight === 0 to know the suite settled.
There is no synchronous balance check. If credit runs out, affected runs come back
with status: "failed" and the reason in each run’s error field — there is no 402.
eval_runs row and return immediately. The response carries a suite_run_id you use to filter the runs back out:
GET /agent-eval/runs?suite_run_id=... and wait for every entry to reach queue_status === "done" before treating the scoring fields as final. The single-case shortcut POST /agent-eval/cases/:id/run blocks up to 60s for synchronous results, but suite-run is always async.
Per-suite-run overrides
Passagent_overrides in the request body to apply a config tweak to every spawned run on top of each case’s own agent_overrides. Useful for “run the whole suite with this draft system_prompt” without touching any case definitions.Authorizations
Your Yappr API key (e.g. ypr_live_...). Generate one in the dashboard under Settings → API Keys.
Path Parameters
Body
Optional overrides applied to every spawned run on top of each case's agent_overrides.