Read a Hugging Face dataset revision into Compute

What this connects

Dataset repositories you read yourself, in one direction. You materialize one split at one revision, and Lithi reads the bytes you staged.

The conformance level is import_compatible: Lithi reads released open file formats under a published conformance matrix. There is no Hub API client here, and Lithi never signs in to a repository for you.

Bind an exact input

Three things identify a dataset read, and all three belong in your records: the repository, the split, and the revision.

A branch name is not a revision. Resolve it to the exact commit before you read, then stage that read and pin its digest:

"snapshot": {
  "version_id": "<repository, split and resolved revision>",
  "content_sha256": "<the digest of the bytes you staged>"
}

A dataset can be updated, gated or withdrawn between your read and your run. The digest is what turns that into a visible refusal instead of a quiet difference in your results. See how to pin a snapshot that cannot change.

Credentials

A private access token stays in your own reading tooling. You use it to fetch the data; Lithi never receives it, and no work request has a field for it.

Where Lithi reads your staged copy, input.credential_ref names a storage credential you registered as a reference. Register it, test it, read it back, and revoke it when the work ends.

Formats and limits

Preserve the features declaration rather than flattening it into untyped columns. Class labels, sequences and image or audio references all lose meaning when they arrive as free text, and a workflow then grades the wrong thing.

Name the released profile in input.format and the item identifier in item_id_field. Where the dataset has no stable identifier, add one before you stage, and keep it identical across every split you run.

Provenance travels with the run. provenance.json in your result bundle binds the input, workflow, model, schema and envelope digests, so a published result stays traceable to the revision it came from. Carry the dataset's own license and citation terms yourself; Lithi does not evaluate them.

Commit the output

Accepted rows land in data/part-* with the namespaced result columns appended, and typed failures, abstentions and preserved disagreements are written beside them.

result-manifest.json is written last and is the only complete-bundle marker. Read the counts there before you publish anything derived from the run.

If you republish the output as a dataset, publish the revision you pinned alongside it. A derived dataset without its source revision is an assertion, not a record.

What is not supported

Lithi makes no wire, SDK or endpoint compatibility claim for any Hub service, and reading an open format does not create one.

Outside the adapter:

• Downloading a repository on your behalf, public or gated.
• Resolving main, a tag or any moving pointer at run time.
• Accepting a token, and therefore any access that depends on one.
• Training, fine-tuning or publishing a model. Model and build assets are a different input path.

If the staged bytes moved after you quoted, admission answers REPLAN_REQUIRED. Re-stage at a current revision, pin the new digest, quote again, then approve again.