Read Compute results from a PyTorch dataset

What this connects

Your own training code, as a reader. You build a dataset over an accepted result bundle, or over the immutable bundle you staged as an input.

The conformance level is import_compatible, with a published conformance matrix for the file formats read. Your training runs on your own hardware; Compute produces the data it reads.

Bind an exact input

A dataset must be built over one frozen bundle, never over a directory that is still filling.

Read result-manifest.json first. It is written last, it is the only complete-bundle marker, and it binds every file, count and schema you are about to iterate.

Pin the same bundle identity in your experiment record:

"snapshot": {
  "version_id": "<the result bundle or staged input identity>",
  "content_sha256": "<the digest of those exact bytes>"
}

If your dataset can reach a file the manifest does not bind, your epochs are not reproducible. See how to pin a snapshot that cannot change.

Credentials

Nothing in your training loop needs a Lithi secret. Download the bundle once, with a download authority you hold, then read from local storage.

Where the loader reads staged storage directly, input.credential_ref names a registered credential reference. Never place a secret in a dataset class, a checkpoint, a log line or a launch script.

Formats and limits

Keep the readers disjoint, and make disjointness a property of the split rather than of chance.

• For a map-style dataset, build the index once from the manifest and split the index across your workers. Every item then has exactly one owner.
• For an iterable dataset, assign whole parts to workers. A part is the natural unit; splitting inside one invites duplicated items.
• Never derive a split from a shuffle seed alone. Two workers with the same seed read the same rows.

Preserve item identity and schema through the reader. Carry item_id_field into every batch, keep identifiers as strings, and take types from the manifest rather than from inference. The namespaced result columns, including lithi_status, lithi_confidence and lithi_verification, are what let you filter or weight a batch honestly.

Commit the output

Write predictions back as a table keyed by the same item identifier, then stage that table as its own immutable bundle if it becomes the input to another run.

Record the bundle digest beside the checkpoint. A checkpoint without the digest of the data that produced it cannot be reproduced, only re-attempted.

Reconcile before you report. Compare the item count you iterated against the manifest count, and account for the failure and abstention counts rather than treating the accepted rows as the whole set.

What is not supported

Compute is not a training platform, and holds no wire, SDK or endpoint compatibility claim about any framework.

Outside the adapter:

• Submitting an arbitrary training script, container or executable to run in Compute. No such input is admitted.
• Distributed training, gradient exchange or checkpoint hosting.
• Reading a bundle that has no result-manifest.json yet. Part files alone never mean the job finished.
• Inferring types or identifiers from filenames.

If a bundle is incomplete, the job reports PARTIAL with a next action. Take the accepted parts, resubmit the outstanding items, and rebuild the index from the newer manifest before you train again.