Run a test suite and collect the evidence

What you get

The evidence a test run produces, kept whole rather than summarized into a verdict.

• A result per test case in data/part-*, with its outcome, duration and the identifier you can search for.
• The build and test logs, and any artifacts the suite wrote, delivered alongside the report.
• Cases that could not run at all in failures/part-*, with a typed code, a safe reason and a next action.
• provenance.json, binding the exact source snapshot, configuration and workflow that produced the run.

A red suite and a suite that never compiled are different outcomes. Keeping them apart is the point of the failure partition.

What you need

A source snapshot and a declaration of exactly what to build and test. Vague scope is the main reason a run like this becomes unreadable.

run_id,scheme,test_target,configuration,platform
APT-01,PaymentsApp,PaymentsUnitTests,Debug,macOS
APT-02,PaymentsApp,PaymentsUnitTests,Debug,iOS Simulator
APT-03,PaymentsApp,PaymentsSlowTests,Debug,macOS

Declare run_id as your item_id_field. Name one test target per row rather than "all tests", so a failure points at a target instead of at the whole project.

Declare the isolation you expect too. The run gets a bounded workspace that carries no state from any other run. Anything your tests need must be in the snapshot or declared as a dependency. A hermetic unit suite is a smaller and far more reproducible request than a suite that expects a live service.

Bring your build inputs deliberately: how model and build assets are accepted.

Run it

1. Inspect locally: lithi inspect ./test-fixture. Confirm the snapshot contains the project, the test target and the dependencies you assumed were there.
2. Read the workflow: lithi workflow explain <workflow-id>, and read which platforms, configurations and timeouts it accepts as parameters.
3. Set a deadline that suits the suite. A test target that hangs should hit a declared deadline rather than run until something else stops it.
4. Quote it: lithi quote --workflow <workflow-id> --input ./test-fixture/manifest.csv. The quote gives a maximum charge and an expiry, and authorizes nothing.
5. Approve the quote separately, then submit with lithi batch submit and track with lithi status <job-id>.

Start with one target on one platform. Adding the second platform is trivial once the first run reads correctly.

Read the result

Open result-manifest.json first. It is written last, so its absence means the run has not finished, whatever else is in the bundle.

Read the failure partition before the test report. A compile failure or a missing dependency invalidates the whole report, and it usually means the snapshot was incomplete rather than that a test regressed.

Then read the per-case results. Check that the count of cases matches what your suite runs locally. A green run that executed far fewer cases than you expected is the failure mode worth hunting, because it looks exactly like success.

Download the logs and artifacts and keep them with the receipt: download a bundle and verify it.

Where it stops

You get a test report and artifacts. You do not get a signed, notarized or published build.

By default the run signs nothing, notarizes nothing and publishes nothing. No archive is uploaded to a distribution channel, no version is tagged, and nothing reaches a store or a customer. Those are separate acts with their own authority, and this workflow never performs them for you.

Nothing is written back into your repository or your continuous integration system either. The bundle is evidence you read and decide on.

A passing suite proves those cases passed on that snapshot in that configuration. It is not a promise about behavior you did not test, and no accuracy or coverage is implied.