Turn a document corpus into cited chunks

What you get

Chunks that still know where they came from.

• One chunk per row in data/part-*, carrying the document identifier, the page range and the heading path above it.
• A citation on every chunk, precise enough that a person can open the source page and find the text.
• Documents that could not be read in failures/part-*, with a typed code, a safe reason and a next action.
• Vector artifacts where the workflow declares them and your data profile permits them.

A chunk without a citation is unusable in an answer. If your assistant cannot show the page, nobody can check it, and an unverifiable answer is worse than no answer.

What you need

Twelve documents and a manifest. Include the formats that break naive chunkers, not just clean prose.

doc_id,path,format,page_count,language
RAG-01,handbook/policy.pdf,pdf,42,en
RAG-02,handbook/appendix-tables.pdf,pdf,11,en
RAG-03,notes/faq.md,markdown,0,en
RAG-04,scans/fax-notice.pdf,pdf,3,en
RAG-05,handbook/policy-es.pdf,pdf,42,es
RAG-06,archive/locked.pdf,pdf,0,en

Declare doc_id as your item_id_field. Document two is mostly tables, document four is a poor scan, and document six is protected and should be refused rather than returning empty pages.

Decide your chunk bounds before you run. Name a target size and an overlap. Then name the splitting rule: split on a semantic boundary such as a heading or paragraph, never mid-sentence.

Run it

1. Inspect locally: lithi inspect ./corpus-fixture. It reports page counts, encryption flags and whether a page carries a text layer at all.
2. Read the workflow: lithi workflow explain <workflow-id>, and read the chunking parameters and the vector artifacts it declares.
3. Set the chunk parameters explicitly, and ask for the heading path and page range to be carried on every chunk. Retrofitting a citation later is not possible.
4. Quote it: lithi quote --workflow <workflow-id> --input ./corpus-fixture/manifest.csv. The quote gives a maximum charge and an expiry, and authorizes nothing.
5. Approve the quote separately, then submit with lithi batch submit and track with lithi status <job-id>.

Prepare the inputs properly first: how documents and media are accepted.

Read the result

Open result-manifest.json, then read chunks rather than counts.

Take ten chunks and read them cold, without the source. Each one should make sense on its own. A chunk that starts mid-clause or ends before the condition that qualifies it will retrieve well and answer badly.

Then check the boundaries you care about. No chunk should span two unrelated headings. A table should arrive as a table, not as a column of numbers with the header stranded in a different chunk. Open the cited page for five chunks and confirm the text is there.

Read the scanned document separately. Lower confidence and more abstentions on a poor scan are correct. The failure to hunt for is a misread figure inside an otherwise fluent chunk.

Where it stops

You get chunks, citations and permitted vector artifacts. Nothing is loaded into your index, your database or your assistant.

Good chunks are a precondition for retrieval, not a substitute for measuring it. Nothing here shows that your assistant will find the right chunk, rank it first, or answer correctly from it. Retrieval quality is a property of your index, your queries and your model. Measure it on your own questions: grade candidates against a held-out set.

No accuracy is promised. Extraction quality depends on the page, and a corpus mixing languages or formats will chunk unevenly. A person reviews the refusals and decides what to fix at the source.