Start an eval run
POST/v1/projects/:project_id/evals/:eval_id/runs
Runs the eval against its dataset, creating one real agent generation per item and scoring the outputs.
wait: true executes the run synchronously and returns it terminal, with its scores. The dataset is capped at 25 items for a synchronous run; a larger one is rejected with 400 rather than partially scored.
wait: false (the default) enqueues one task per item and returns immediately with status: "queued". A worker executes the items and the run settles itself; poll GET /evals/{eval_id}/runs/{eval_run_id} for the terminal status. There is no item cap on a queued run.
The whole run is pinned to one agent version, stamped on agent_version: pass one explicitly to evaluate a canary before promoting it, or omit it to use the active release's stable version (or the live draft when no release is in effect). Without the pin, release assignment would bucket each item independently and blend two configs into a single score.
With baseline_run_id, the finished run's aggregate_scores.baseline carries per-scorer deltas against that run, computed over the items present and scorable in both runs, with the divergence counted. A delta over a shifted dataset is therefore never presented as a clean comparison.
Request
Responses
- 201
- 400
- 401
- 403
- 404
- 409
- 413
- 429
- 500
- 502
- 4XX
Eval run finished (wait: true) or queued (wait: false)
Bad request (non-boolean wait, dataset empty or over the synchronous cap, unknown agent_version, invalid baseline, scorers no longer valid against the agent, a tool_context key that cannot become a header)
Unauthorized
Forbidden
Eval not found
PROJECT_PAUSED — the project is paused, so nothing new starts in it; error.meta carries paused_at and pause_reason.
REQUEST_REJECTED — the request body is over the size the body parser accepts
QUOTA_EXCEEDED — a requests quota on the calling API key or its project is spent for the window; Retry-After says when it reopens
Response Headers
Seconds until the quota window admits a request again
Internal server error
The deployment's admission service is asked about this write and did not answer — ADMISSION_UNAVAILABLE; the write did not happen
The deployment's admission service refused this write — answered under the service's own 4xx status, with its code (matching ^[A-Za-z_][A-Za-z0-9_]{0,63}$, not necessarily an ERROR_CODES key), message and meta; the write did not happen. A status this operation also declares on its own is documented there