Create an eval
POST/v1/projects/:project_id/evals
Binds an agent under test to a dataset and a list of scorers. The agent and the dataset must belong to the same project as the eval; a cross-project reference is rejected with 400.
Scorer config is frozen here rather than read from the agent at run time, so two runs of the same eval are always judged by the same criteria and their comparison measures the agent instead of the config drifting underneath it. Each scorer type may appear at most once.
Request
Responses
- 201
- 400
- 401
- 403
- 404
- 409
- 413
- 429
- 500
- 502
- 4XX
Eval created successfully
Bad request (unknown scorer type, cross-project reference, invalid threshold)
Unauthorized
Forbidden
The project does not exist or the caller is not a member of it (RESOURCE_NOT_FOUND).
An eval with that name already exists in the project
REQUEST_REJECTED — the request body is over the size the body parser accepts
QUOTA_EXCEEDED — a requests quota on the calling API key or its project is spent for the window; Retry-After says when it reopens
Response Headers
Seconds until the quota window admits a request again
Internal server error
The deployment's admission service is asked about this write and did not answer — ADMISSION_UNAVAILABLE; the write did not happen
The deployment's admission service refused this write — answered under the service's own 4xx status, with its code (matching ^[A-Za-z_][A-Za-z0-9_]{0,63}$, not necessarily an ERROR_CODES key), message and meta; the write did not happen. A status this operation also declares on its own is documented there