Skip to main content

Ingestion Rules

How a file type becomes readable text — the converter a project reaches for, and the chunking that follows.

Overview​

Document ingestion can read PDFs, plain text and Markdown on its own. An ingestion rule is how a project teaches it everything else: match a MIME type glob, name a converter — a tool or an agent — and every matching file is handed to that converter and the text it returns is chunked, embedded and indexed.

The rule is also where a project's chunking defaults live, so a corpus is configured once instead of on every ingest call.

This module is a verbatim mirror of the runtime: every field, method, status code and error shape is the runtime's own, re-rooted under the project in the path.

One answer is naturali's own: changing or deleting one of the rules naturali deploys for managed conversion answers 403 MANAGED_RESOURCE_READ_ONLY, on every path the runtime writes through, not only these routes. Only those rules are read-only; a rule of your own stays writable whatever its metadata says.

See the OpenAPI spec for the full endpoint and schema reference, or browse it rendered under API Reference → Ingestion Rules.

Data Model​

IngestionRule​

FieldTypeDescription
policy_idsstring[]The policies every conversion the rule starts acts under. Empty for one written without any, which does not run until a write gives it one.
idstring
project_idstring
content_type_globstring
tool_idstring, nullable
agent_idstring, nullable
actionstring, nullable
preset_parametersobject, nullable
native_extractionfirst | skip
file_deliverybase64 | download_url
chunk_strategystring, nullable
chunk_sizeinteger, nullable
chunk_overlapinteger, nullable
metadataobjectCaller-owned annotations on a resource, stored as the object they were written as: the types a value was written with are the types a read returns, so a filter can ask an ordering question about a number. Unlike other body fields, keys are stored and returned verbatim in the casing supplied — they are not converted between snake_case and camelCase. No key is reserved, and that is the point: every piece of state the platform owns lives in its own typed column, so nothing written here reaches platform state. The platform never reads the bag — it is not an IAM context, not a policy input and not part of a prompt — which is what separates it from a tag bag.
created_atstring (date-time)
updated_atstring (date-time)

content_type_glob is required: a MIME glob matched against a file's content_type (image/*, application/vnd.openxmlformats-*). The converter is a tool (tool_id) or an agent (agent_id), never both; action names the operation, required for an mcp tool converter, and preset_parameters are merged into a tool converter's input before every invocation. native_extraction first (the default) converts only when built-in extraction yields no text; skip always converts. file_delivery is how the file reaches a tool converter: base64 (the default) or download_url. chunk_strategy, chunk_size and chunk_overlap are the project's defaults, overridable per ingest request.

Key Concepts​

Matching a file​

content_type_glob is matched against the uploaded file's content_type — so the glob is only as good as the type the upload declared, which is why Answer from images and audio sends the real one. image/* catches every image; application/pdf catches exactly one type.

One converter, either kind​

Exactly one of tool_id or agent_id must be set; setting both, or neither, is a 400. The two differ in what they are good at:

  • A tool converter is a deterministic call — an OCR service, a spreadsheet-to-text endpoint. preset_parameters pins the arguments that are the same every time, and file_delivery decides whether the tool receives the bytes inline as base64 or a URL to fetch. An mcp tool also needs action, naming which of its operations to call.
  • An agent converter is a model reading the file — useful when the source needs interpretation rather than parsing, like describing a diagram. The managed conversion rules are agent converters; see one read a photo.

When the built-in extractor is skipped​

native_extraction decides who goes first for a type ingestion can already read. first — the default — tries built-in extraction and only calls the converter when that yields nothing, which is what you want for a PDF that is mostly text but occasionally a scan. skip always calls the converter, which is what you want when the converter is strictly better than the built-in path.

Chunking defaults​

chunk_strategy, chunk_size and chunk_overlap on a rule are the project's defaults for files that rule matches; the same fields on an ingest request override them per call. Setting them here means a corpus is configured in one place, not repeated in every ingest.

Conversion authority​

A rule runs its converter under its own policy_ids, capped by whoever ingests the document. A converter that grant may not start (tools:CallTool or agents:CreateAgentGeneration) is never called and the ingest fails; every tool call a converter agent makes is held to the grant too. A rule with empty policy_ids converts nothing until a write gives it some.

policy_ids is required on create (400 VALIDATION_FAILED). The writer must hold every grant the policies carry (403 POLICY_GRANT_NOT_HELD), and a policy a rule names cannot be deleted (409 POLICY_HAS_DEPENDENTS).

Who may do what​

Every route needs any project member.

Examples​

Route images through an OCR tool​

pol_V1StGXR8Z5jdHi6B allows tools:CallTool on the converter (Conversion authority).

naturali create-ingestion-rule \
--project-id proj_V1StGXR8Z5jdHi6B \
--content-type-glob "image/*" \
--tool-id tool_V1StGXR8Z5jdHi6B \
--policy-ids pol_V1StGXR8Z5jdHi6B \
--file-delivery base64 \
--chunk-strategy whole

List the project's rules​

naturali list-ingestion-rules --project-id proj_V1StGXR8Z5jdHi6B

Always convert, never extract natively​

naturali update-ingestion-rule \
--project-id proj_V1StGXR8Z5jdHi6B \
--ingestion-rule-id igr_V1StGXR8Z5jdHi6B \
--native-extraction skip

Delete a rule​

naturali delete-ingestion-rule \
--project-id proj_V1StGXR8Z5jdHi6B \
--ingestion-rule-id igr_V1StGXR8Z5jdHi6B