Build and activate a scoring rubric

Before you start

You need three things. A written definition of good, a reference set whose right answers you already know, and the workflow whose output the rubric will judge.

Without the reference set you can create a rubric, but you cannot calibrate it. That distinction runs through this whole page.

Write the three parts

A rubric has hard gates, scored dimensions and evidence rules. They do different jobs, and mixing them is the most common mistake.

1. Hard gates are pass or fail conditions. An item that fails a gate is rejected whatever it scored elsewhere.
2. Scored dimensions are the qualities you weigh. Each needs a scale and a plain definition of what each point means.
3. Evidence rules say what a check must cite from the item to support its score.

Keep gates few and unambiguous. A gate that needs interpretation belongs as a scored dimension instead.

Version it, never edit it

Rubrics are versioned. You create a version, you version it again to change it, and you retire versions you no longer want activated.

The version that judged your work is recorded in verification.json, and the digests behind the run are recorded in provenance.json. A pinned version is what makes two runs comparable at all.

Editing a rubric in place would quietly change what past results meant. That is why the lifecycle does not offer it.

Calibrate before you activate

Creating a version and activating it are separate acts, and the gap between them is where the work happens.

Run the version against your reference set first. Compare its verdicts with the answers you already know. If it disagrees with you on items you are certain about, the rubric is the thing to fix.

Look at the disagreements one by one. Usually a dimension is underdefined, or a gate is doing a scored dimension's job.

Activate through the custom evaluation preset

A calibrated version becomes live when your request selects it through the Custom evaluation quality preset, or through the profile your team applies.

Read back what is actually activated before you rely on it. The activated version is a fact about your account, not about the file you last edited.

Abstention and disagreement are normal

An item with insufficient evidence abstains, and lands under abstentions/. Two independent checks that differ leave a preserved disagreement under disagreements/.

Neither is a defect. Both are information about where your rubric is under strain.

A rubric that never abstains on genuinely hard items is usually one that guesses. Treat a zero abstention rate as a question, not a result.

If a rubric is not behaving

Read the disagreements before you change anything, then re-read the gates that produced the rejections.

Fix it by creating a new version and calibrating that, then retire the old one. Every non-success outcome carries a safe reason and a next action, so start from what the result already told you.

For what each preset proves, see quality presets. For reading the three non-accepted outcomes, see failures, abstentions and disagreements.