Skip to main content

Gate a rollout on an eval

By the end of this tutorial you will have a staged rollout that refuses to promote until an eval run pinned to the new version reads supported — and a new version that went live with that run recorded beside it.

It joins two things you may already know: Roll out an agent version (releases) and Score an agent change (evals). Neither is repeated here; this page is about the gate between them.

Ten steps:

  1. Create a dataset.
  2. Add the cases the new version must pass.
  3. Create the eval.
  4. Write the new version.
  5. Start a gated rollout.
  6. Try to promote — refused.
  7. Run the eval without a version — it measures the old one.
  8. Run the eval against the new version.
  9. Promote.
  10. Read the evidence on the version.

Every step is one API call, shown for all three clients. The ids in the responses are examples — copy the ones your own calls return.

Prerequisites​

  • A nat_sk_… API key exported as NATURALI_TOKEN, and a client set up as in Create a provider.
  • The agent from Your first agent generation, with PROJECT, PROVIDER and AGENT exported. Its instructions are Answer in one sentence. If you are unsure, say so. — it knows no refund policy yet.
export NATURALI_TOKEN=nat_sk_...
export PROJECT=proj_V1StGXR8Z5jdHi6B
export PROVIDER=aip_V1StGXR8Z5jdHi6B
export AGENT=agent_V1StGXR8Z5jdHi6B

The steps assume the agent is at version 1 with no rollout running. If you already did Roll out an agent version, use its current version as stable_version in step 5 and the version step 4 returns as canary_version.

This tutorial spends model budget

Each eval run generates once per case and the judge grades once per case: sixteen model calls in total for the two runs below.

1. Create a dataset​

naturali create-dataset \
--project-id "$PROJECT" \
--name refund-policy \
--description 'Answers the refund policy must keep right'
{
"id": "dset_SQlIIzyfjZ8RfnWp",
"project_id": "proj_V1StGXR8Z5jdHi6B",
"name": "refund-policy",
"description": "Answers the refund policy must keep right",
"created_at": "2026-10-03T11:07:51.183Z",
"updated_at": "2026-10-03T11:07:51.183Z"
}
export DATASET=dset_SQlIIzyfjZ8RfnWp

2. Add the cases the new version must pass​

while IFS='|' read -r question answer; do
naturali create-dataset-item \
--project-id "$PROJECT" \
--dataset-id "$DATASET" \
--input "[{ \"role\": \"user\", \"content\": \"$question\" }]" \
--expected-output "$answer"
done <<'CASES'
What is our refund window?|Refunds are accepted within 30 days of purchase.
How many days do I have to ask for a refund?|You have 30 days from purchase to ask for a refund.
Can I get a refund 10 days after buying?|Yes, refunds are accepted within 30 days of purchase.
Can I get a refund 45 days after buying?|No, refunds are accepted only within 30 days of purchase.
CASES

Each call answers the item it created; the first:

{
"id": "dsit_QXulqVULqGNKh7ar",
"dataset_id": "dset_SQlIIzyfjZ8RfnWp",
"input": [{ "role": "user", "content": "What is our refund window?" }],
"expected_output": "Refunds are accepted within 30 days of purchase.",
"source_generation_id": null
}

3. Create the eval​

The eval is what the rollout will name as its gate. Its pass_threshold: 0.5 is read against the 95% interval of the pass rate, so the run must show more than half the cases pass: four of four is the fewest that clears it. An llm_judge scorer grades each answer against expected_output and needs a model of its own, here ai_provider_id.

Every run starts the agent under the eval's policy_ids, so first write the policy it runs under: it allows starting a generation and nothing else.

naturali create-policy \
--project-id "$PROJECT" \
--name refund-gate-run \
--description "What each run of the refund-gate eval may start." \
--document "{
\"statement\": [
{ \"effect\": \"Allow\", \"action\": [\"agents:CreateAgentGeneration\"], \"resource\": [\"srn:$PROJECT:*:*\"] }
]
}"
export POLICY=pol_Hk4nT7wQ2cYv8RxL

Then create the eval:

naturali create-eval \
--project-id "$PROJECT" \
--name refund-gate \
--agent-id "$AGENT" \
--dataset-id "$DATASET" \
--policy-ids "$POLICY" \
--pass-threshold 0.5 \
--scorers '[{ "type": "llm_judge", "prompt": "Rate 0-1 how well the answer matches the reference. Answer with {\"score\": <0-1>, \"reasoning\": \"<why>\"}. Question: {{input}} Answer: {{output}} Reference: {{expected}}", "pass_threshold": 0.7, "ai_provider_id": "'"$PROVIDER"'" }]'
{
"id": "eval_RPMTrH6wp2eHPrC1",
"name": "refund-gate",
"agent_id": "agent_gCmLeRABaJYLeM2Z",
"dataset_id": "dset_SQlIIzyfjZ8RfnWp",
"policy_ids": ["pol_Hk4nT7wQ2cYv8RxL"],
"scorers": [
{
"type": "llm_judge",
"prompt": "Rate 0-1 how well the answer matches the reference. …",
"ai_provider_id": "aip_CLcDNRe8GpP1pBNs",
"pass_threshold": 0.7
}
],
"pass_threshold": 0.5
}
export EVAL=eval_RPMTrH6wp2eHPrC1

4. Write the new version​

Give the agent the policy. The write archives version 2; nothing serves it yet.

naturali patch-agent \
--project-id "$PROJECT" \
--agent-id "$AGENT" \
--instructions 'Answer in one sentence. Our refund policy: refunds are accepted within 30 days of purchase. If you are unsure, say so.'
{
"id": "agent_gCmLeRABaJYLeM2Z",
"instructions": "Answer in one sentence. Our refund policy: refunds are accepted within 30 days of purchase. If you are unsure, say so.",
"version": 2,
"active_release": null
}

5. Start a gated rollout​

PUT /v1/projects/{project_id}/agents/{agent_id}/release with promotion_gate set to the eval. Traffic splits exactly as it would without a gate — the gate only decides how the rollout may end.

naturali set-agent-release \
--project-id "$PROJECT" \
--agent-id "$AGENT" \
--stable-version 1 \
--canary-version 2 \
--canary-percent 10 \
--promotion-gate "$EVAL"
{
"id": "agent_gCmLeRABaJYLeM2Z",
"version": 2,
"active_release": {
"stable_version": 1,
"canary_version": 2,
"canary_percent": 10,
"promotion_gate": "eval_RPMTrH6wp2eHPrC1"
}
}

The gate must be an eval of this agent in this project; anything else is a 400.

6. Try to promote​

naturali promote-agent-release \
--project-id "$PROJECT" \
--agent-id "$AGENT"

409:

{
"error": {
"code": "PROMOTION_GATE_UNMET",
"message": "Promotion gate 'eval_RPMTrH6wp2eHPrC1' has no completed eval run against version 2 of agent 'agent_gCmLeRABaJYLeM2Z'.",
"meta": {
"promotion_gate": "eval_RPMTrH6wp2eHPrC1",
"agent_version": 2
}
}
}

The rollout keeps running untouched: 10% of traffic still gets version 2.

7. Run the eval without a version​

A run that names no agent_version measures the release's stable version — the one already serving most traffic.

naturali start-eval-run \
--project-id "$PROJECT" \
--eval-id "$EVAL" \
--wait true
{
"id": "evrun_4IKmAxJsIFNWDsba",
"eval_id": "eval_RPMTrH6wp2eHPrC1",
"agent_version": 1,
"status": "completed",
"aggregate_scores": {
"scorers": { "llm_judge": { "mean": 0.1, "pass_rate": 0 } },
"pass_rate": 0,
"scored_item_count": 4,
"pass_rate_interval": { "low": 0, "high": 0.49, "level": 0.95 }
},
"verdict": "refuted"
}

agent_version: 1: this run says nothing about the candidate. Even had it read supported, it could not open the gate — only a run pinned to the canary version counts.

8. Run the eval against the new version​

Pin the run with agent_version. The whole run uses that one configuration, whatever the traffic split.

naturali start-eval-run \
--project-id "$PROJECT" \
--eval-id "$EVAL" \
--wait true \
--agent-version 2
{
"id": "evrun_T8ZROCSa4dTvZ7H4",
"eval_id": "eval_RPMTrH6wp2eHPrC1",
"agent_version": 2,
"status": "completed",
"aggregate_scores": {
"scorers": { "llm_judge": { "mean": 0.8, "pass_rate": 1 } },
"pass_rate": 1,
"scored_item_count": 4,
"pass_rate_interval": { "low": 0.51, "high": 1, "level": 0.95 }
},
"verdict": "supported"
}

status: completed, verdict: supported, agent_version: 2 — what the gate checks on the newest run of this agent against the canary. A newer run that reads insufficient or refuted closes the gate again.

9. Promote​

The same call as step 6.

naturali promote-agent-release \
--project-id "$PROJECT" \
--agent-id "$AGENT"
{
"id": "agent_gCmLeRABaJYLeM2Z",
"instructions": "Answer in one sentence. Our refund policy: refunds are accepted within 30 days of purchase. If you are unsure, say so.",
"version": 2,
"active_release": null
}

Version 2 now serves all traffic.

10. Read the evidence on the version​

GET /v1/projects/{project_id}/agents/{agent_id}/versions shows which run cleared the gate.

naturali list-agent-versions \
--project-id "$PROJECT" \
--agent-id "$AGENT"
{
"data": [
{
"id": "agver_m7PsncJPs9ePvgbu",
"version": 2,
"label": null,
"eval_run_id": "evrun_T8ZROCSa4dTvZ7H4"
},
{
"id": "agver_ldzzwXGEy09kNAvK",
"version": 1,
"label": null,
"eval_run_id": null
}
],
"next_cursor": null
}

eval_run_id on version 2 is the run from step 8. Anyone reading the history later can open that run and see what the version was measured against.

What's next​