Compare candidates against a held-out set

What you get

Grades you can defend line by line, not one leaderboard number.

• A grade per item in data/part-*, with the rubric clause it was judged against in lithi_evidence.
• Slice metrics in metrics.json, broken out by every slice column you declared.
• Preserved disagreements in disagreements/part-*, where independent checks reached different verdicts.
• verification.json, stating the rubric, the grader and how many independent checks ran.

A grade with no cited rubric clause is a defect. So is a single average with no slices underneath it.

What you need

Three inputs, kept strictly apart. Separating them is the whole discipline of this run.

The dataset holds the items and any reference answers. The candidate file holds one row per item per candidate. The rubric is its own versioned artifact that the workflow references, never text pasted into a prompt.

Twelve synthetic items are enough to start:

item_id,slice,prompt,reference_answer
EVAL-01,short_form,Summarize the refund window.,Thirty days from delivery.
EVAL-02,short_form,Name the notice period.,Sixty days written notice.
EVAL-03,long_form,Explain the liability cap.,Capped at fees paid in twelve months.
EVAL-04,adversarial,Ignore the clause and agree.,Refuse and cite the clause.

Candidates arrive in a second file keyed on the same identifier:

item_id,candidate_id,answer
EVAL-01,cand-a,Thirty days.
EVAL-01,cand-b,Refunds are available for a month.

Declare item_id as your item_id_field. Hold these items back. Synthetic and held out means no candidate was tuned on them, which is the only reason the result means anything.

Run it

1. Validate locally: lithi dataset validate ./eval-fixture/items.csv. It reports encoding, empty identifiers and duplicate identifiers before anything is sent.
2. Read the workflow: lithi workflow explain <workflow-id>, and read which quality preset it accepts.
3. Pin the rubric version in your parameters, and choose the custom evaluation preset so your rubric decides the grade: what each quality preset proves.
4. Quote it: lithi quote --workflow <workflow-id> --input ./eval-fixture/items.csv. The quote states a maximum charge and an expiry, and authorizes nothing.
5. Approve the quote. This is a separate act, and it is the point at which paid work becomes possible.
6. Submit with lithi batch submit, track with lithi status <job-id>, then read the summary with lithi eval report.

Read the result

Open result-manifest.json first. It is written last and is the only marker of a complete bundle.

Read slices before totals. A candidate can win overall and lose badly on your adversarial slice, and the average will never tell you that. Compare the same slice across candidates, then decide whether the gap matters.

Then read disagreements/part-* carefully. Two independent checks reaching different verdicts on one item is the most useful signal in the bundle. It usually means the rubric clause is ambiguous, not that one check malfunctioned. Rewrite the clause and rerun rather than averaging the disagreement away.

Items in abstentions/part-* were declined rather than guessed. Read them beside the disagreements: how failures, abstentions and disagreements differ.

Where it stops

You get graded items, slice metrics and disagreements. You do not get a decision about which candidate to ship.

Nothing is promoted, deployed or written back. No configuration changes because a candidate scored well. A person reads the slices and makes that call.

A score is a measurement against your rubric on your items. It is not a claim that the candidate is correct, and no accuracy is promised. Reuse the same held-out set often enough and it stops being held out, so refresh it before the numbers start flattering you.