Skip to main content
POST
Create case
A case is the runnable unit of agent eval — persona + agent + scenario + success criteria. Suites are optional; a case with suite_id: null is an ad-hoc case you can run on its own.

​
Worked example

​
Assertion shape

Each entry in success_criteria is one assertion with these fields:
An assertion missing its type/kind, or a must_say / must_not_say missing both phrase and pattern, fails with an explicit reason naming the missing field — it is never treated as a silent pass.

​
Choosing tool_policy

Under every policy the invocation is recorded, so must_call_tool passes on a tool that was held back — a mocked run still proves your agent tried. The assembled request body lands on the run’s turn as tool_calls.args, which is what must_call_tool’s args_match compares against:
tool_name is the tool’s name — not the flow step’s name, which can differ, and not a reformatted variant of it. GET /agent-eval/runs/{id}/turns shows the tool name, the arguments and whether the call dispatched: under mock or allowlist each recorded call carries dispatched — false for a request that was assembled but never sent, true for one that really fired — for webhook tools and connected-app steps alike. Under real the field is absent, because nothing could have held the call back. tool_allowlist entries match a tool’s id or its name, exactly and case-sensitively. Prefer ids: tool names are not unique within a workspace, so allowlisting a name arms every tool sharing it — possibly pointing at a different endpoint than the one you meant. An entry that matches nothing is mocked rather than fired, so a typo fails safe (and shows in the trace as dispatched: false).
allowlist currently differentiates webhook tools only. Steps that call a connected app (Google Calendar, Gmail) still run for real under allowlist, because app connections are gated separately and have no per-tool name to match against. Use mock if you need those held back too.
What mock does and does not prove. It proves your agent decides to call the tool, and passes the right arguments. It does not prove your endpoint is reachable or that it accepts the request — target validation only happens on a real dispatch, so a tool pointed at an unreachable or rejected URL passes a mocked run and fails on the first real call.Because the synthetic response body is fixed ({"success": true}), a step whose transitions branch on the content of the response can never match under mock — a mocked step always takes its success edge. Run those with real (or allowlist) to exercise the response-conditional branch.
tool_policy applies to both agent types. A flow agent’s tool steps and connected-app steps, and the tools attached to a single-prompt agent, all follow the table above — a prompt agent’s tool sends the same request a live call sends under real, and returns the same synthetic success under mock.Two behaviours are policy-independent, because they make no request to hold back: a hang-up tool ends the test run, and a transfer tool ends it without ever placing a call to the destination.

Authorizations

​
Authorization
string
header
required

Your Yappr API key (e.g. ypr_live_...). Generate one in the dashboard under Settings → API Keys.

Body

application/json
​
name
string
required
Example:

"Yes path — agreement on first ask"

​
agent_id
string<uuid>
required
​
persona_id
string<uuid>
required
​
scenario
string
required
​
description
string
​
suite_id
string<uuid> | null
​
success_criteria
object[]
​
max_turns
integer
default:20
Required range: 1 <= x <= 100
​
pass_threshold
number
default:80
Required range: 0 <= x <= 100
​
agent_overrides
object | null
​
tool_policy
enum<string>
default:mock
Available options:
mock,
real,
allowlist
​
tool_allowlist
string[]

Response

Case created

A specific eval scenario — persona + target agent + scenario + success criteria.

​
id
string<uuid>
required
​
company_id
string<uuid>
required
​
agent_id
string<uuid>
required

Agent under test. Full agent record is expanded inline as agent in API responses.

​
persona_id
string<uuid>
required
​
name
string
required
Example:

"Yes path — caller agrees on first ask"

​
scenario
string
required

Free-form one-paragraph framing the persona LLM is given on top of its identity. Describe the situation that prompted the call.

Example:

"The persona is responding to a missed call from your business about their recent inquiry. They have time to talk for 5 minutes."

​
success_criteria
object[]
required

Array of assertions evaluated after the run completes.

​
max_turns
integer
default:20
required

Hard cap on conversation turns. Hitting this terminates the run with termination_reason='max_turns'.

Required range: 1 <= x <= 100
​
pass_threshold
number
default:80
required

Weighted-score threshold (0-100) for pass_fail=true.

Required range: 0 <= x <= 100
​
tool_policy
enum<string>
default:mock
required

How the agent's tools behave during the run. Applies to BOTH agent types — a flow agent's tool_call steps and the tools attached to a single-prompt agent, which the model calls directly. mock (default): webhook tools make no request — each returns the fixed synthetic result {"success": true, "status_code": 200}. The canonical request body is still assembled (so argument-resolution bugs still surface, and must_call_tool + args_match still assert against it), but target validation and the network hop are skipped, so mock does not prove the endpoint is reachable or acceptable. Because the synthetic body is fixed, a step whose transitions branch on the response CONTENT always takes its success edge under mock. real: tools fire for real (charges real money, hits real systems). allowlist: webhook tools matching an entry in tool_allowlist fire for real, the rest return the same synthetic success. Note that allowlist differentiates webhook tools ONLY — steps calling a connected app (Google Calendar, Gmail) dispatch for real under both real and allowlist, because those are gated separately and have no per-tool name to match on. Use mock to hold those back too. Hang-up and transfer tools are policy-independent: they make no request, so in a run they simply end it (a transfer never places a call to its destination).

Available options:
mock,
real,
allowlist
​
created_at
string<date-time>
required
​
agent
object
​
persona
object

Reusable caller archetype consumed by eval cases. The identity_prompt plus behavior_traits shape how the persona LLM responds; the same persona can be reused across many cases.

​
suite_id
string<uuid> | null

Optional parent suite. When null, the case is ad-hoc — runnable on its own but not part of a regression sweep.

​
description
string | null
​
agent_overrides
object | null

Optional per-case overrides applied to the agent's saved config at run time (e.g. a different system_prompt or flow_config for A/B testing). Same shape as the agent record. The agent on disk is never mutated. Configuration only: the identity fields id, company_id, company_timezone, tools and flow_tools are stored but ignored at run time — a run always executes as the agent's own workspace, with the tools that workspace resolved.

​
tool_allowlist
string[]

Used only when tool_policy='allowlist'. Entries match a tool's id or its name, exactly and case-sensitively. Prefer ids: tool names are not unique within a company, so a name arms every tool that shares it. An entry that matches nothing is mocked rather than fired — a typo fails safe, and shows in the run trace as dispatched: false.

​
updated_at
string<date-time>
​
deleted_at
string<date-time> | null