Skip to main content

Documents

The retrievable unit of knowledge: a source, split into embedded chunks, ready for knowledge search.

Overview​

A document is text an agent can be given at answer time. It comes from one of two places — a string you post directly, or a file already stored in the project — and in both cases the same thing happens next: the source is split into chunks, each chunk is embedded, and the document becomes searchable.

For a posted string that work is synchronous — the call answers 201 with the document already indexed. For a file it is background by default: the ingest call answers 202 immediately with status: pending and a document id, and the chunking and embedding happen behind it. Pass wait=true on an ingest when you would rather block and get a ready document back.

This module is a verbatim mirror of the runtime: every field, method, status code and error shape is the runtime's own, re-rooted under the project in the path.

A plan caps how much your account stores

Free stores 1 GB, Pro 30 GB, Business 100 GB; an Enterprise ceiling is set by contract. Ingesting past that answers 403 PLAN_LIMIT_REACHED with resource: "storage", and the plan, the limit and what your account is holding as used in meta. This holds however the write arrives: a call to this API, a formation deploy or an agent's tool call.

The ceiling is your account's, not each project's: every project the account pays for draws on the same figure, so one project may hold all of it. Read where you stand from storage on GET /v1/users/me/billing; the usage route below reports one project at a time.

The gigabytes are indexed storage, not the size of your files. A document is stored as the source file plus one row per chunk, and every chunk carries an embedding vector of about 4 kB whatever its text. So the same corpus counts for between two and seven times its own size depending on chunk_strategy — whole and page-based chunking count least, small size windows most. Your current figure is the gb_day component of GET /v1/projects/{project_id}/usage under meter_type=storage.

Reading, editing metadata and deleting stay open at the ceiling — only the routes that add bytes are refused: a create, an ingest, a PATCH carrying content and a version restore, since each re-embeds and keeps the old version. Deleting documents is how an account gets back under its limit. The figure is sampled once a day, so a deletion frees room at the next sample rather than immediately.

See the OpenAPI spec for the full endpoint and schema reference, or browse it rendered under API Reference → Documents.

Data Model​

DocumentRecord​

FieldTypeDescription
idstringDocument ID
file_idstringUnderlying file ID
project_idstringProject ID
pathstring, nullableLogical path of the document within the project (e.g. /reports/q1.txt)
filenamestringOriginal filename
content_typestringMedia type of the source file the document was ingested from. Absent when the underlying file is gone.
sizeintegerFile size in bytes
titlestringDocument title. Absent when none was set.
metadataobjectCaller-owned annotations. Absent when none were set.
tagsobjectThe document's tag bag. Absent when none were set.
statuspending | processing | ready | failed | withdrawnIngestion lifecycle state. pending — enqueued; processing — chunks being extracted and embedded; ready — fully indexed; failed — processing error (see the error field on GET /documents/{id}/status); withdrawn — taken out of listings and search, with its chunks dropped, and restorable from an earlier version.
versionintegerThe document's content version, starting at 1. Every write that changes the content or its annotations archives the state it replaced and increments this.
contentstring, nullableText content (only present on getDocument, and only when status is ready)
chunk_strategypage | whole | sizeThe chunk strategy the document was last (re-)ingested with. Absent when the default (whole) was used — the mapper omits the key rather than sending null.
chunk_sizeintegerWindow size in characters used when chunk_strategy=size. Absent otherwise.
chunk_overlapintegerOverlap in characters between consecutive windows used when chunk_strategy=size. Absent otherwise.
created_atstring (date-time)
updated_atstring (date-time)

An ingestion response adds chunk_count, the chunks created from the source.

DocumentStatusRecord​

What the status endpoint returns — a progress view, not the document.

FieldTypeDescription
idstringDocument ID
statuspending | processing | ready | failedIngestion lifecycle state.
chunk_countintegerNumber of chunks currently indexed for this document (a live count). Grows while status=processing and equals the final total once ready; 0 while pending.
total_chunksinteger, nullablePlanned total number of chunks, known once chunking begins. null until then. Used as the denominator for progress.
total_pagesinteger, nullableNumber of source pages extracted. Only known after extraction, so it is null until status is ready or failed (not the same as zero pages).
progressinteger, nullableIngestion progress as a percentage (chunk_count / total_chunks). 0 while pending, climbs while processing (capped at 99), 100 when ready, and null when failed or not yet computable.
errorobject, nullableWhy the ingestion failed, when status is failed; its code is for example FILE_PARSE_FAILED or INGESTION_TIMEOUT.

Key Concepts​

Two ways in​

POST /v1/projects/{project_id}/documents takes the text itself in content. Use it for anything you already hold as a string — a scraped page, a database row, a support macro.

POST /v1/projects/{project_id}/documents/ingest takes a file_id and parses the stored file (Answer from your documents ingests a Markdown file this way). PDFs are read page by page; text/plain and text/markdown are read whole. Scanned PDFs, images and audio are converted for you — see Managed conversion. Anything else needs an ingestion rule naming a converter.

A file backs one document: a second ingest of the same file_id answers 409 FILE_ALREADY_INGESTED. To re-chunk what you already ingested, use POST /v1/projects/{project_id}/documents/{document_id}/ingest — it discards the existing chunks and runs again against the same source, which is also how a document stuck in failed is recovered. To index the same source twice under different paths, upload a second copy of the file.

Managed conversion​

Every project converts these with no setup:

Content typeConverted when
application/pdfThe PDF has no text layer (a scan)
image/*Always, into one chunk
audio/*Always, transcribed into one chunk

naturali keeps three ingestion rules in the project for this, with the policy they convert under, deployed as the naturali-conversion formation. You can list them, but changing or deleting the rules, the policy or the formation answers 403 MANAGED_RESOURCE_READ_ONLY. A rule of your own on a more specific type, such as image/png, wins over the managed image/*. A type you already had a rule for when conversion was deployed is left to your rule. Answer from images and audio ingests a photo and a recording and reads back the text conversion wrote.

Each conversion is a generation recorded in your project, and it is billed like any other generation on a model naturali provides, except that its whole input is billed at the conversion rate, whatever the media type. A project whose credit balance is negative cannot ingest.

To stop it, set managed_conversion: false with PATCH /v1/projects/{project_id}. This needs the admin role. The managed rules are removed and your own stay. A scanned PDF, an image or an audio file then fails to ingest unless one of your rules matches it. Setting managed_conversion: true turns conversion back on.

naturali update-project \
--project-id proj_V1StGXR8Z5jdHi6B \
--managed-conversion false

The ingestion lifecycle​

pending → processing → ready, or failed. The two ingest routes are background by default, answering 202 Accepted with status: pending; wait=true makes them block and answer 201 Created with status: ready. Creating from a posted string has no such toggle — it indexes inline and answers 201.

Background is the right default for a file — a large PDF outlives a sensible request timeout, and wait=true on one that is too big is a 413 telling you to retry without it. Poll GET /v1/projects/{project_id}/documents/{document_id}/status for progress: it is a small response built for polling, and its progress is what to show a user. A document is only returned by knowledge search once it is ready. Answer from your documents waits for it before the first search.

Chunking​

chunk_strategy decides how the source is cut up, and therefore what a search result looks like:

  • page — one chunk per non-empty page. The default for file ingestion, and the one that keeps a citation meaningful for a PDF.
  • whole — the entire source as one chunk. The default for a posted string. Best when the text is already short enough to hand to a model entirely.
  • size — fixed character windows of chunk_size (default 1000) overlapping by chunk_overlap (default 200). The overlap is what stops a sentence that straddles a boundary from being unfindable.

An ingestion rule can set the project's default; a request may always override it.

Tags and metadata​

GET / PUT / PATCH …/documents/{document_id}/tags read, replace and merge a document's tag map — PUT drops the keys you leave out, PATCH only adds and overwrites the ones you send. Tags travel into search results, which is what makes them useful for filtering a corpus by team, language or source.

A metadata schema declares what a directory's documents must carry in metadata; a write that violates it is refused.

Put a label in tags: a flat map of strings, matched exactly and returned with every search hit. Put the record in metadata: any JSON, kept with its types, so a number orders as a number and a range or in filter applies. A field you compare (revision >= 3) belongs in metadata. A resource that carries only metadata, such as a task, keeps every field there.

Versions, withdrawal and relations​

Every write that changes the content or its annotations archives a version. GET …/versions lists them newest first, and POST …/versions/{version}/restore writes one back as a new version, re-chunking and re-embedding the content. A version carries created_by_principal_type and created_by_principal_id: the user, or for key auth the project's key the request ran on, which is not the key the caller sent.

POST …/withdraw takes a document out of listings and search while keeping its history, archived as a version with no content; its status reads withdrawn, and restoring a content version brings it back. It does not apply under /.system/. DELETE is the permanent act: it removes the document and its source file.

A document can assert typed edges about others (derived_from, supersedes, cites) through its …/relations routes; the edge belongs to the document it leaves.

Exporting the corpus​

GET /v1/projects/{project_id}/documents/export streams the project's documents as newline-delimited JSON, oldest first. It returns what the listing returns for the same caller, minus withdrawn documents and the /.system/ root; path_prefix narrows it.

Who may do what​

Every route needs any project member.

Examples​

Create a document from text​

naturali create-document \
--project-id proj_V1StGXR8Z5jdHi6B \
--content "Refunds are issued within 5 business days." \
--path /policies/refunds.txt \
--title "Refund policy"

Ingest an uploaded file, page by page​

naturali ingest-document \
--project-id proj_V1StGXR8Z5jdHi6B \
--file-id file_V1StGXR8Z5jdHi6B \
--chunk-strategy page \
--path-prefix /handbook/

Watch the ingestion finish​

naturali get-document-status \
--project-id proj_V1StGXR8Z5jdHi6B \
--document-id doc_V1StGXR8Z5jdHi6B

Re-chunk a document that is already ingested​

naturali reingest-document \
--project-id proj_V1StGXR8Z5jdHi6B \
--document-id doc_V1StGXR8Z5jdHi6B \
--chunk-strategy size \
--chunk-size 800 \
--chunk-overlap 150

Delete a document​

naturali delete-document \
--project-id proj_V1StGXR8Z5jdHi6B \
--document-id doc_V1StGXR8Z5jdHi6B