Run Compute over an Iceberg or Hudi table snapshot

What this connects

Open table formats you already manage, in one direction. Lithi reads one table state you named and writes an accepted result bundle you commit back yourself.

The conformance level is import_compatible. Only the format and commit profiles with a published conformance matrix are read, and that matrix states the supported upstream version, fields, lifecycle, errors and limits.

Bind an exact input

A table snapshot is not the same object as the files under it, and the difference matters here more than anywhere else.

The snapshot or instant is the table's own statement of which files belong to which state. The Parquet files beneath it may be rewritten by compaction while the snapshot stays valid, or may be expired away entirely.

So pin both, in snapshot:

"snapshot": {
  "version_id": "<the Iceberg snapshot id or the Hudi instant>",
  "content_sha256": "<the digest of the file set that state resolves to>"
}

version_id records what the table calls that state. content_sha256 records what you actually read. If a maintenance job rewrote the files, the two stop agreeing and the run refuses rather than reading something else. See how to pin a snapshot that cannot change.

Credentials

input.credential_ref names a storage credential you registered. It is a reference, never a secret, and it grants only the read you intended.

Catalog credentials stay in your own tooling. Lithi does not authenticate to a metastore or catalog service on your behalf, so nothing in a work request needs to carry that access.

Formats and limits

Name the released profile in input.format, and check it before you rely on a partition column, a nested field or a decimal scale. A field the profile does not support is refused by name, not silently coerced.

Retention matters more than usual with these tables. If your snapshot expiry runs before your job does, the state you pinned can stop resolving. Give the run a deadline in limits that sits well inside your retention window.

Commit the output

Accepted rows are written to data/part-* with the namespaced result columns appended, and result-manifest.json is written last as the only complete-bundle marker.

Committing that output into a table is your own act. Write the accepted parts as a new snapshot or instant, then reconcile the new state's row count against the manifest before you let readers see it.

Where you registered a destination instead, delivery reaches VERIFIED before COMMITTED, and reports COMMIT_CONFLICT rather than overwriting a state that moved. Retry the delivery; the job does not rerun. See how immutable commits work.

What is not supported

Lithi is not a table format engine and makes no wire, SDK or endpoint compatibility claim about one.

Outside the adapter:

• Reading a table by asking a catalog or metastore to resolve it for you.
• Format or commit profiles with no published conformance matrix.
• Incremental, changelog or time-travel reads across a range of states. Pin one state per job.
• Compaction, clustering, expiry or any other table maintenance.

If the pinned state stops resolving before admission, you get REPLAN_REQUIRED with a next action. Pin a current snapshot, quote again, and approve again.