Compute
Walk through a sandbox job end to end
Open the guided sandbox scenario, read a mock quote, run a small synthetic classification set, then inspect the accepted rows, the failure and the abstention before resetting.
What this walkthrough covers
You will run one small evaluation job from quote to result without spending anything. The whole run is synthetic: no account, no credentials, no live work and no charge. It takes a few minutes, and you can reset and repeat it as often as you want.
The set is deliberately tiny, at eight rows, so you can read every outcome by hand.
Open the guided scenario
lithi sandbox open --scenario ai-eval-team-v1
The scenario gives you a synthetic input, a workflow, and a sandbox context to run inside. Everything it creates is marked synthetic and is discarded on reset.
The input is eight short product reviews, each with an expected sentiment label. It is addressed as a synthetic location rather than one of your sources:
synthetic://evals/v1
Two rows are there on purpose to fail cleanly. One carries text the workflow cannot parse. One is genuinely ambiguous, so a careful grader should decline to answer rather than guess.
Read the mock quote and approve it
Ask for a quote for the same workflow before running anything. The mock quote states a maximum charge, an expiry, and that it authorizes nothing on its own.
A quote never starts work, in the sandbox or live. Approving it is a separate act you take after reading it. Approve it now to continue.
Run the job and watch it change state
lithi batch --sandbox --workflow eval.semantic_grade.v1 --input synthetic://evals/v1
The job reports its state as it goes:
ADMITTED— the request was accepted against the approved quote.RUNNING— items are being graded.PARTIAL— the run finished with a mix of accepted and non-accepted items.
PARTIAL is the expected ending here, not a fault. It is what a real run looks like when most items succeed and a few do not.
Open the sample result bundle
lithi result --sandbox <job-id>
The sample bundle has the same shape as a live one:
data/part-* six accepted rows
failures/part-* one typed failure
abstentions/part-* one abstained row
metrics.json aggregate and slice metrics
verification.json rubric and grader evidence
provenance.json input, workflow and schema digests
receipt.json sample receipt projection
result-manifest.json written last, binds every file and count
Open data/part-* first. Each row carries lithi_status, lithi_result, lithi_confidence and lithi_evidence beside your own columns.
Then read the two rows that did not land in data/. The failure row carries a lithi_error_code, a safe reason and a next action. The abstained row carries no answer at all, because the grader declined rather than guessing under the quality preset in force.
Check the three things that matter
Confirm result-manifest.json is present. It is written last and is the only marker that a bundle is complete; part files on their own never mean the run finished.
Then confirm the counts add up to eight across data/, failures/ and abstentions/. Finally, confirm the receipt is labeled a sample, so you never mistake it for evidence about live work.
Reset, then decide what is next
lithi sandbox reset
Reset discards the job, the mock quote and the bundle, and returns the scenario to its first step. Run it again to try a different fixture, or read what each one is teaching you.
Every outcome you just saw has a live counterpart. The fixture catalog lists all eight and what each one teaches you to recognize.
When you want a real result instead of a sample, see moving to live work.