Prepare a Hugging Face dataset

Choose a permitted revision and split

Your application selects the dataset revision and split, and your agent keeps the task goal, context and final decision. A compatible client or Kit helper can carry the fixed export when the selected workflow provides one. Lithi accepts the approved artifact reference and returns a result that you reconcile with the original item IDs.

Use your existing Hugging Face environment to choose the dataset repository, configuration, exact commit and split you are allowed to process. Record the library version as well. A branch name can move, so use a fixed revision for reproducible work. Hugging Face documents the revision parameter and local loading methods in its loading guide.

Check the dataset's access terms and your permission to process its contents. Keep gated-dataset tokens in your own credential store. Do not give a Compute job a Hub token or assume a public repository grants every downstream use.

Inspect a local synthetic export

This example requires the datasets package. It creates two local records, writes a temporary JSONL file and verifies the round-trip. It doesn't download a Hub dataset or contact Lithi.

import hashlib
import json
from pathlib import Path
from tempfile import TemporaryDirectory
from datasets import Dataset
rows = [{"item_id": "sample-001", "text": "Hello"},
        {"item_id": "sample-002", "text": "Goodbye"}]
dataset = Dataset.from_list(rows)
with TemporaryDirectory(prefix="lithi-hf-example-") as folder:
    path = Path(folder) / "records.jsonl"
    dataset.to_json(str(path))
    payload = path.read_bytes()
    restored = [json.loads(line) for line in payload.splitlines()]
    assert restored == rows
    print({"rows": len(restored), "sha256": hashlib.sha256(payload).hexdigest()})

The expected row count is two. The digest identifies the bytes produced by your installed encoder; it isn't a Compute artifact ID. The Dataset API reference describes from_list and to_json.

Preserve the selected dataset's meaning

Keep stable item IDs before filtering or shuffling. Record the exact split, filter and any transformation. Keep training, validation and held-out evaluation populations separate. A dataset fingerprint is useful local provenance, but retain the immutable source revision and exact exported files as well.

For streaming data, define a finite reviewed selection and materialize it before quoting work. Do not submit a moving iterator as a fixed input. Review any decoding or transformation code in your own environment; dataset content does not authorize arbitrary code execution on a Compute provider.

Hand off and verify the result

Select only fields and formats accepted by the chosen workflow. Freeze byte counts and digests, then follow file preparation. The native request uses an encrypted artifact reference, not a dataset repository name or access token. Check availability and review the quote before approving paid work.

Lithi processes the admitted task. On return, verify the result and reconcile accepted, failed and missing items through the original ID mapping. Review exceptions before adding labels or generated text to another dataset. A successful export doesn't establish a training or model-hosting integration.

Recover the reviewed selection

If a revision or gated file is unavailable, repair access or review a new selection; don't switch to the current branch silently. If a local output load fails after Compute completes, reload the existing accepted result. Keep result delivery and deliberately new work separate.

What should a small test prove?

Use a permitted fixed selection to check that its IDs and selected fields survive export and that you can join reviewed outcomes back to those records. The local example above checks serialization; the first test walks through the separate task and result review.

Next, prepare typed records with dataframe tools.