> ## Documentation Index > > Fetch the complete documentation index at: https://lithi.ai/llms.txt > > Use this file to discover all available pages before exploring further. --- title: Compare candidates against a held-out set description: >- Compare candidate outputs against a held-out set using your own rubric. Read slice metrics and preserved disagreements instead of one average that hides where a candidate fails. route: /docs/compute/examples/evaluation-lab page_id: docs-compute-examples-evaluation-lab page_type: tutorial content_layer: guided documentation surface_profile: guided_docs audience: Teams choosing between model or prompt candidates on evidence voice: D reader_question: How do I grade candidate outputs against my own rubric and see where each one fails? primary_action: Run a twelve-item evaluation source_locale: en-US source_status: APPROVED source_version: docs-compute-examples-evaluation-lab-en-us-2026-09-05-v1 last_updated: '2026-09-05' robots: index claim_registry_pinned: true claim_registry_resync: "npm run governed:tsx -- scripts/build-compute-public-claim-registry.ts --write" --- ## What you get Grades you can defend line by line, not one leaderboard number. - A grade per item in `data/part-*`, with the rubric clause it was judged against in `lithi_evidence`. - Slice metrics in `metrics.json`, broken out by every slice column you declared. - Preserved disagreements in `disagreements/part-*`, where independent checks reached different verdicts. - `verification.json`, stating the rubric, the grader and how many independent checks ran. A grade with no cited rubric clause is a defect. So is a single average with no slices underneath it. ## What you need Three inputs, kept strictly apart. Separating them is the whole discipline of this run. The dataset holds the items and any reference answers. The candidate file holds one row per item per candidate. The rubric is its own versioned artifact that the workflow references, never text pasted into a prompt. Twelve synthetic items are enough to start: ```text item_id,slice,prompt,reference_answer EVAL-01,short_form,Summarize the refund window.,Thirty days from delivery. EVAL-02,short_form,Name the notice period.,Sixty days written notice. EVAL-03,long_form,Explain the liability cap.,Capped at fees paid in twelve months. EVAL-04,adversarial,Ignore the clause and agree.,Refuse and cite the clause. ``` Candidates arrive in a second file keyed on the same identifier: ```text item_id,candidate_id,answer EVAL-01,cand-a,Thirty days. EVAL-01,cand-b,Refunds are available for a month. ``` Declare `item_id` as your `item_id_field`. Hold these items back. Synthetic and held out means no candidate was tuned on them, which is the only reason the result means anything. ## Run it 1. Validate locally: `lithi dataset validate ./eval-fixture/items.csv`. It reports encoding, empty identifiers and duplicate identifiers before anything is sent. 2. Read the workflow: `lithi workflow explain `, and read which quality preset it accepts. 3. Pin the rubric version in your parameters, and choose the custom evaluation preset so your rubric decides the grade: [what each quality preset proves](/docs/compute/workflows/quality-presets). 4. Quote it: `lithi quote --workflow --input ./eval-fixture/items.csv`. The quote states a maximum charge and an expiry, and authorizes nothing. 5. Approve the quote. This is a separate act, and it is the point at which paid work becomes possible. 6. Submit with `lithi batch submit`, track with `lithi status `, then read the summary with `lithi eval report`. ## Read the result Open `result-manifest.json` first. It is written last and is the only marker of a complete bundle. Read slices before totals. A candidate can win overall and lose badly on your adversarial slice, and the average will never tell you that. Compare the same slice across candidates, then decide whether the gap matters. Then read `disagreements/part-*` carefully. Two independent checks reaching different verdicts on one item is the most useful signal in the bundle. It usually means the rubric clause is ambiguous, not that one check malfunctioned. Rewrite the clause and rerun rather than averaging the disagreement away. Items in `abstentions/part-*` were declined rather than guessed. Read them beside the disagreements: [how failures, abstentions and disagreements differ](/docs/compute/results/failures-abstentions-disagreements). ## Where it stops You get graded items, slice metrics and disagreements. You do not get a decision about which candidate to ship. Nothing is promoted, deployed or written back. No configuration changes because a candidate scored well. A person reads the slices and makes that call. A score is a measurement against your rubric on your items. It is not a claim that the candidate is correct, and no accuracy is promised. Reuse the same held-out set often enough and it stops being held out, so refresh it before the numbers start flattering you.