Skip to main content

Evaluations

Repeatable test suites — datasets of cases, scorers that grade them, and runs that record what an agent version scored.

Overview​

Three resources, in the order you meet them:

  • a dataset is a named collection of test cases (a case is the input messages to replay, plus an optional reference answer);
  • an eval binds one agent to one dataset and freezes the list of scorers that grade it;
  • an eval run executes the eval — one real generation per item, against exactly one agent version — and records per-item results plus the aggregate scores.

So the loop is: curate cases, bind them to an agent, run before you promote, and compare against the previous run.

This module is a verbatim mirror of the runtime: every field, method, status code and error shape is the runtime's own, re-rooted under the project in the path. It serves two collections — /datasets and /evals — under that project prefix.

On every plan

Evals are not gated by plan. Establishing that an agent works on your data is part of deciding whether to buy, so a suite runs on Free as on Enterprise. What still applies is what applies to any generation: a run's items count against the plan's monthly runs, and its managed tokens against the balance.

See the OpenAPI spec for the full endpoint and schema reference, or browse it rendered under API Reference → Evaluations.

Data Model​

Dataset​

FieldTypeDescription
idstring
project_idstring
namestring
descriptionstring, nullable
created_atstring (date-time)
updated_atstring (date-time)

name is unique within the project; description says what the suite covers.

DatasetItem​

FieldTypeDescription
idstring
dataset_idstring
inputobject[]Messages replayed verbatim as the generation's input
expected_outputstring, nullable
metadataobjectCaller-owned annotations on a resource, stored as the object they were written as: the types a value was written with are the types a read returns, so a filter can ask an ordering question about a number. Unlike other body fields, keys are stored and returned verbatim in the casing supplied — they are not converted between snake_case and camelCase. No key is reserved, and that is the point: every piece of state the platform owns lives in its own typed column, so nothing written here reaches platform state. The platform never reads the bag — it is not an IAM context, not a policy input and not part of a prompt — which is what separates it from a tag bag.
source_generation_idstring, nullableThe generation this item was curated from. A curated item is a deliberate fixture: erasing the source generation neither deletes nor mutates it.
created_atstring (date-time)
updated_atstring (date-time)

expected_output is the reference answer the scorers compare against. json_logic and tool scorers can read metadata.

Eval​

FieldTypeDescription
policy_idsstring[]The policies every run of the eval acts under. Empty for one written without any, which does not run until a write gives it one.
idstring
project_idstring
namestring
agent_idstring
dataset_idstring
scorersobject[]Scorer configs, a discriminated union on type. Each type may appear at most once. Every scorer produces { score: 0–1, passed: boolean }; binary scorers emit 0 or 1. exact_match compares the trimmed output text to expected_output. contains looks for value in the output text. json_logic evaluates expression over { input, output, object, expected, item.metadata }, where object is the structured output (absent when the agent has no output_schema). output_schema validates the structured output against the scorer's own schema, falling back to the agent's; it requires the agent to carry an output_schema, because without one the platform emits no structured output and every item would score 0. llm_judge grades the output with a model completion, returning a continuous score plus its reasoning. Its pass_threshold is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. embedding_similarity embeds the output text and expected_output with the platform's configured embedding model (EMBEDDING_PROVIDER / EMBEDDING_MODEL — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its pass_threshold is required for the same reason as the judge's. An item without an expected_output scores 0; an embedding backend failure marks the item errored, never a score of 0. tool runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct name — outcomes and aggregates key on the name. decider grades each item with a decision of a project decider: its score expression reads the decision's answers. Like tool, it keys on its name and may appear several times.
pass_thresholdnumber, nullable
group_bystring, nullableThe item metadata key a run rolls its scores up by; null when the eval groups nothing
evidenceobject, nullableThe rules a run's verdict reads beyond the threshold. Null declares none.
claimobject, nullableWhat a run of the eval tests: the statement a supported verdict backs, the population its dataset stands for, and the limits outside which the verdict says nothing. Each run copies it when created. Null states none.
created_atstring (date-time)
updated_atstring (date-time)

name is unique within the project. pass_threshold (0–1) gates a run: its verdict is supported when the 95% interval of the pass rate sits at or above the threshold, refuted when it sits below, and insufficient when it holds the threshold or nothing was scored, so a small suite cannot read as supported on a lucky point estimate. evidence.required_groups names group_by values each judged the same way on its own interval; the run's verdict is the worst of them, and verdict_reasons says why, one entry per scope. null reports scores without a verdict. scorers are described under Scorers.

EvalRun​

FieldTypeDescription
idstring
eval_idstring
agent_idstring, nullableThe agent every item of this run executed against, resolved when the run was created and kept after that agent is deleted or the eval is pointed at another. Null for a run that recorded none, which satisfies no promotion gate.
agent_versionintegerThe one agent version every item in this run executed against
definitionobject, nullableThe eval's scorers, pass threshold, grouping key and evidence as they were when the run was created. Every item is scored and the run settled by these; a later edit to the eval reaches only runs created after it. Null when the run recorded none.
claimobject, nullableThe eval's claim as it was when the run was created: what this run tested. A later edit to the eval reaches only runs created after it. Null when the eval stated none.
decider_versionsobject, nullableThe decider version each decider scorer grades under, keyed by scorer name and resolved when the run started. Null when the eval has no decider scorer.
statusqueued | running | completed | failed | cancelled
baseline_run_idstring, nullable
trigger_idstring, nullableThe trigger that started this run — set when a schedule (or a manual trigger fire) started it, null for a run started through this API. Kept even if the trigger is later deleted.
aggregate_scoresobject, nullablePer-scorer rollup plus the run-level pass rate; null until the run is terminal
verdictsupported | refuted | insufficient, nullablesupported when the pass rate's 95% interval sits at or above definition.pass_threshold, refuted when it sits below, and insufficient when it holds the threshold or nothing was scored; the worst of that and every required group. Null when the run has no threshold, and until it completes.
verdict_reasonsobject[], nullableOne entry per judged scope, the run first. Null when verdict is.
item_countinteger
completed_countinteger
errored_countinteger
metadataobjectThe caller-owned key/value metadata supplied when the run was started, returned verbatim. Absent when the run was started without any (a trigger-started run included — see trigger_id for that provenance). The server writes nothing here.
started_atstring (date-time), nullable
finished_atstring (date-time), nullable
created_atstring (date-time)

id is the polling handle; baseline_run_id names the run this one is compared against. A run's tool_context is write-only — see tool_context on an eval run.

EvalResult​

FieldTypeDescription
idstring
eval_run_idstring
dataset_item_idstring, nullableNull once the dataset item has been deleted
inputobject[]Messages replayed verbatim as the generation's input
expected_outputstring, nullable
generation_idstring, nullable
outputstring, nullableThe agent's final output text. Cleared when the linked generation's content is purged; the scores and the frozen input survive.
scoresobject[]
passedbooleanAND over the per-scorer passed flags
errorobject, nullableItem-level failure reason. A generation that did not complete — a requires_action pause (GENERATION_NOT_COMPLETED), a provider failure — or a scorer that could not grade the output is recorded here and excluded from the aggregates rather than scored 0.
created_atstring (date-time)

scores holds one { scorer, score, passed, reasoning } entry per scorer.

Key Concepts​

Scorers​

Every scorer produces { score: 0–1, passed: boolean }, and each type may appear at most once per eval:

TypeGrades by
exact_matchtrimmed output text equals expected_output
containsoutput text contains value (case_sensitive optional)
json_logica JSON Logic expression over input, output, object, expected and item.metadata
output_schemathe structured output validates against a JSON Schema — requires the agent to carry an output_schema
embedding_similaritycosine similarity between the embeddings of the output text and expected_output, clamped to 0–1
llm_judgea model grades the output against a prompt; returns a continuous score plus its reasoning
toola tool of yours scores the item and returns { score, passed?, reasoning? }

llm_judge requires a pass_threshold: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. Pin its ai_provider_id and model when you care about comparability — two runs judged by different models are not comparable. Score open-ended answers writes a rubric judge and reads its reasoning.

embedding_similarity requires a pass_threshold for the same reason. It grades meaning rather than characters, so a reworded-but-correct answer scores well where exact_match scores 0. An item with no expected_output scores 0; if the embedding call itself fails the item is marked errored rather than scored 0, so an outage cannot read as a fleet of failing answers.

A tool scorer may appear several times under distinct names, and the tool must be server-callable: an http, mcp or pipeline tool. A client tool pauses for a caller that an eval run does not have.

Scorer config is frozen on the eval​

Scorers are stored on the eval rather than read at run time, so two runs of the same eval are judged by identical criteria and their difference measures the agent instead of the config moving underneath it. Changing the scorers is an explicit PUT /v1/projects/{project_id}/evals/{eval_id}. Score an agent change freezes an llm_judge and a gate this way.

A claim says what a verdict backs​

An eval's optional claim states what its runs test: the statement a supported verdict backs, the population its dataset stands for, and the limits outside which the verdict says nothing. Each run copies the claim when it is created, so a later edit reaches only later runs, and a gated rollout's 409 names it beside the verdict.

A debt cancels a run in flight​

An eval run generates against the agent under test, so POST /v1/projects/{project_id}/evals/{eval_id}/runs answers 402 INSUFFICIENT_CREDIT while the project owner owes for usage already served. A run that was already going when the debt appeared is not refused — it is cancelled, by the same reconciliation that found the debt, on every project that owner pays for.

A queued or running run is cancelled; a run that has finished, failed or was already cancelled is left alone. Start it again after a top-up: a cancelled run keeps whatever results it had recorded, and the cancellation is recorded too.

Only a negative balance does this — a zero balance runs. See A negative balance stops managed generation.

Datasets are fixtures, not views​

A dataset item is a copy. Curating one with POST /v1/projects/{project_id}/datasets/{dataset_id}/items/from-generation promotes a real completed generation into a test case — its input becomes the item's input, its own answer becomes expected_output unless you supply one — and the item keeps working after that generation's content is purged. Erasing production data can never quietly stop a suite from being runnable. Score an agent change curates one from an answer the agent got wrong.

Only a completed generation can be promoted, and only while its content is still available: an agent running with trace_content_mode: none never stored the input, so there is nothing to copy.

Running a suite on a schedule​

A trigger with target_type: eval is what runs a suite without anyone asking — nightly, or on whatever cron you give it. The run it produces carries that trigger's id, so a regression the schedule caught is attributable to it.

Run authority​

A run acts under the eval's policy_ids, capped by whoever starts it: the caller, or the trigger's policy_ids for a scheduled run. Every item's generation and every scorer run under that authority. A run whose agent, scorer tool or scorer decider the eval's grant may not start is refused 403 ACTION_NOT_HELD before it is recorded, even when its starter may start it. The starter also needs the agent's, each scorer tool's and each scorer decider's start action besides evaluations:RunEval, else those items fail FORBIDDEN. An eval with empty policy_ids scores nothing until a write gives it some.

policy_ids is required on create (400 VALIDATION_FAILED). The writer must hold every grant the policies carry (403 POLICY_GRANT_NOT_HELD), and a policy an eval names cannot be deleted (409 POLICY_HAS_DEPENDENTS).

Runs are background by default​

POST /v1/projects/{project_id}/evals/{eval_id}/runs answers 201 with status: "queued" and settles itself; poll GET /v1/projects/{project_id}/evals/{eval_id}/runs/{eval_run_id} until the status is terminal, then read aggregate_scores. There is no item cap on a queued run. Score open-ended answers starts one and polls it to a verdict.

Pass wait: true to hold the request open and get the terminal run with its scores; a synchronous run is capped at 25 items and a larger dataset is rejected with a 400 rather than partially scored. Score an agent change runs a small suite this way.

A negative balance stops a run before it starts​

A suite generates once per item and again to grade it, so POST /v1/projects/{project_id}/evals/{eval_id}/runs answers 402 INSUFFICIENT_CREDIT and starts nothing while the project owes for usage already served. It is the project owner's balance, and it is read whichever provider the eval's agent runs on — a run resolves its models as it goes, so there is nothing to check them against here. See A negative balance stops managed generation.

A Free account that has used its plan's monthly runs answers 403 PLAN_LIMIT_REACHED with resource: "runs" here as well, on its own credential too. See A plan's run allowance can stop generation.

A create naming a managed model clears the one-source check. Each refusal holds however the write arrives: a call to this API, a formation deploy or an agent's tool call.

Cancelling a run is never refused this way. Neither is creating the eval; only starting one spends.

tool_context on an eval run​

tool_context on run creation is forwarded to every item's generation, so an agent whose tools authorize through tool_context is scored against the configuration it actually runs in production instead of an empty bag. It is stored on the run and re-read per item, since a queued run is driven by a worker with no request behind it, and it is write-only: no read of the run returns it, unlike metadata — a run is a report other people read, and a credential in it is not theirs to see. It is cleared once the run reaches a terminal state.

An eval generation has no session, so the reserved keys session_id, actor_id and actor_external_id are dropped rather than forwarded. Every other key must be a valid HTTP header name, or the run is rejected with 400 INVALID_TOOL_CONTEXT_KEY before anything is created.

One agent version per run​

A run is pinned to a single agent version, stamped on agent_version: pass one explicitly to evaluate a canary before promoting it, or omit it to use the active release's stable version (the live draft when no release is in effect). Without the pin, release assignment would bucket each item independently and blend two configs into one score.

A run also records its agent_id and definition (scorers, pass_threshold, group_by) when it is created, and is scored and settled by them: an edit to the eval, or pointing it at another agent, reaches only later runs. A promotion gate accepts only a run whose agent_id is the agent being promoted.

Comparing against a baseline​

With baseline_run_id, the finished run's aggregate_scores.baseline carries per-scorer deltas against that run — computed over the item intersection, the items scorable in both runs, with added_item_count and removed_item_count reporting the divergence. A delta over a shifted dataset is never presented as a clean comparison. Score an agent change reads one after an instruction fix.

Cancelling a run​

POST /v1/projects/{project_id}/evals/{eval_id}/runs/{eval_run_id}/cancel drops the outstanding items so the run stops spending provider budget and settles as cancelled; a run settles once, so a cancel racing the run's own settlement is 400. Results already written are kept — they measure generations that were really paid for — while aggregate_scores stays null: a partial roll-up in the field a completed run uses would read as a whole-dataset verdict. Cancelling needs evaluations:CancelEvalRun; evaluations:RunEval does not grant it. Score open-ended answers cancels one right after starting it.

Who may do what​

Every route needs any project member.

Examples​

Create a dataset and curate a case from production​

naturali create-dataset \
--project-id proj_V1StGXR8Z5jdHi6B \
--name billing-regressions \
--description "Questions the billing agent regressed on"

naturali create-dataset-item-from-generation \
--project-id proj_V1StGXR8Z5jdHi6B \
--dataset-id dset_V1StGXR8Z5jdHi6B \
--generation-id gen_V1StGXR8Z5jdHi6B

Add a hand-written case​

naturali create-dataset-item \
--project-id proj_V1StGXR8Z5jdHi6B \
--dataset-id dset_V1StGXR8Z5jdHi6B \
--input '[{ "role": "user", "content": "When is my invoice issued?" }]' \
--expected-output "On the first of each month." \
--metadata '{ "topic": "billing" }'

Create an eval with two scorers and a gate​

pol_V1StGXR8Z5jdHi6B allows agents:CreateAgentGeneration on the agent (Run authority).

naturali create-eval \
--project-id proj_V1StGXR8Z5jdHi6B \
--name billing-regression-suite \
--agent-id agent_V1StGXR8Z5jdHi6B \
--dataset-id dset_V1StGXR8Z5jdHi6B \
--policy-ids pol_V1StGXR8Z5jdHi6B \
--pass-threshold 0.8 \
--scorers '[
{ "type": "contains", "value": "invoice" },
{
"type": "llm_judge",
"prompt": "Rate 0-1 how well the answer matches the reference. Answer with {\"score\": <0-1>, \"reasoning\": \"<why>\"}. Question: {{input}} Answer: {{output}} Reference: {{expected}}",
"pass_threshold": 0.7,
"model": "gpt-4o-mini"
}
]'

Run a canary version against the last run​

naturali start-eval-run \
--project-id proj_V1StGXR8Z5jdHi6B \
--eval-id eval_V1StGXR8Z5jdHi6B \
--agent-version 4 \
--baseline-run-id evrun_V1StGXR8Z5jdHi6B

Poll the run, then read the failures​

naturali get-eval-run \
--project-id proj_V1StGXR8Z5jdHi6B \
--eval-id eval_V1StGXR8Z5jdHi6B \
--eval-run-id evrun_9fJk2LmNpQrStUvW

naturali list-eval-results \
--project-id proj_V1StGXR8Z5jdHi6B \
--eval-id eval_V1StGXR8Z5jdHi6B \
--eval-run-id evrun_9fJk2LmNpQrStUvW