Lithi Data & Evaluation

Start with the comparison decision

For a simple comparison, freeze candidates A and B against evaluation set E-01–E-03 and rubric version A. Return one outcome per expected tuple, keep missing and abstained items in the denominator, and inspect the disagreement before choosing a candidate.

Your agent defines the population and rubric for comparing supplied candidate outputs. It prepares the permitted dataset, candidates and stable identities. Lithi returns item-level outcomes, coverage and disagreements. Your agent checks the denominator and supporting evidence before deciding what the comparison means.

Your agent keeps the evaluation question, population and final choice. The Kit prepares the supported comparison request; participating Macs process the bounded candidates, and your agent reviews item coverage and evidence.

Model assets, candidate outputs, evaluation sets, and rubrics belong to the Models & Evaluation information family. A dataset or candidate file is the input; an operation determines whether the result is normalization, a leakage check, scoring, or comparison.

The operation is the exact supported capability. Agent guidance explains the process; it does not select a model or grant access. The Data & Evaluation pack covers normalize-dataset, deduplicate-dataset, classify-batch, prepare-eval-set, leakage-check, evaluate-candidates, compare-candidates, and regression-eval.

Conceptual task brief: Compare candidates A and B on the frozen evaluation set E-01–E-03 under rubric version A. The permitted evidence is each candidate output, the declared reference or its absence, and the rubric. Preserve missing and abstained items in the denominator. Return item-level outcomes, slice coverage, disagreements, and the rubric identity. Accept only when every expected tuple is accounted for and no aggregate score hides missing coverage.

Your integration maps the brief to the selected evaluation operation's request and result contract. The comparison supplies evidence for your decision; it does not automatically promote a model.

Freeze the population and rubric

Pin the original-byte snapshot, logical schema, stable ID columns, grouping and split rules, candidate outputs, rubric, evaluator, and requested result format as separate inputs. Keep reference labels away from candidate generation when the evaluation policy requires a held-out set.

When a dataset or candidate output carries personal or secret content, local preparation resolves to blocked, needing review, or eligible before anything is sealed for transfer. Preserve the stable IDs this page requires inside that same preparation policy.

1. Normalize types and values without changing stable IDs. Record canonical-to-original maps and validation exceptions.
2. Group exact or related duplicates before assigning splits when leakage matters. Keep uncertain matches unresolved.
3. Define the expected relation, such as sample_id × candidate_id × rubric_id, before processing.
4. Request exact output columns. Put item outcomes, provenance, and evidence in declared sidecars instead of surprise _lithi_* columns.
5. Preserve the distinction between a missing member, JSON null, an empty string, and an empty list or object.

Compare what was supplied

compare-candidates and evaluate-candidates inspect existing outputs against a declared rubric. They do not generate a candidate, test a model, edit a dataset, reveal held-out labels, or promote a model. Candidate generation and model testing need their own operation, input contract, and evidence. Inspect item-level results, slice denominators, abstentions, failures, missing outputs, exclusions, and unresolved cases before reading an aggregate score.

This fixture illustrates the expected relation, not a measured score or fixed service schema. Keep IDs such as "000123" as strings. A score summary is incomplete without its population and evaluator identity.

For a regression evaluation, bind baseline, candidate, dataset, rubric, evaluator, and item-level deltas. A new split, correction, rubric, candidate, or conclusion creates a new version. Statistical intervals need appropriate sampling assumptions; a high point estimate alone is not a release decision.

Recover missing coverage

When a row fails schema validation, retain its original identity and exception. When a candidate output is absent, recover the existing artifact or report the missing tuple. Do not score an omitted row as success, drop it from a denominator, or rerun the whole job because one file import failed.

Worked example

Compare candidates on a declared evaluation

Freeze this three-item dataset, two candidate outputs, and rubric version A:

The rubric awards one point for each correct response, including preserving missing evidence. A scores 2/3 and B 3/3; both are 1/1 on arithmetic and missing-evidence handling, and they differ on the condition slice. These are arithmetic summaries of authored examples, not measured accuracy or an automatic promotion. Preserve the item-level disagreement and its rubric clause before choosing a candidate.

Read Text & Data for generic structured operations and Code & Tests for source or test evidence. Keep the item-level comparison as evidence, then use First test to review a small approved sample before expanding the evaluation.

Compare published prices

Take scoring 1,000,000 model answers against a rubric, at 2,000 input tokens and 100 output tokens each.

Published list prices checked 5 October 2026.

Each provider counts tokens with its own tokenizer, so the same text can be a different number of tokens. Rows without a note apply each published rate to the same token counts. These tables compare published prices for the same work, not output quality.

The table compares published prices for the same work, not output quality.