Compute
Submit Compute work from Ray or Spark
Commit immutable inputs before you submit, so retries and speculative execution cannot duplicate paid work, and read accepted output independently of your runtime's partitions.
What this connects
Your own distributed runtime, as the caller. Your job prepares an input, submits Compute work through the Work API, and later reads the accepted bundle back.
The conformance level for the data exchange is import_compatible, backed by a published conformance matrix. There is no registered Ray or Spark runtime adapter, so Compute is never a compute backend for your cluster.
Bind an exact input
Do the freezing before submission, in a stage that runs once. This is the whole design: a task that submits must be able to run twice without changing what it asks for.
Write the input, take its digest, and let the digest travel with the task:
"snapshot": {
"version_id": "<the write identity your job recorded>",
"content_sha256": "<the digest of those exact bytes>"
}
Never build the request inside a task that a retry could re-execute against fresh bytes. If the bytes can move, the digest moves, and a retry stops being a retry. See how to pin a snapshot that cannot change.
Credentials
input.credential_ref names a credential you registered as a reference. Do not broadcast a secret to your workers, and do not put one in a task closure, a driver log or an environment variable your runtime prints.
A work key carries the six work scopes only. Management scopes are never mintable on it, so a submitting job cannot change identities, destinations or finance settings.
Formats and limits
Name a released profile in input.format and a stable identifier in item_id_field. Item identity is what makes a partial result reconcilable, and your runtime's partition numbers are not identity.
The maximum item count for a workflow is stated on its descriptor and repeated on your quote. Set deadline and maximum_price in limits, and remember that a quote never authorizes work.
Commit the output
Retries and speculative execution are the reason this page exists, and the protection is idempotency rather than luck.
Every effect carries an idempotency tuple and a payload digest, in separate scopes: request, batch, manifest, logical item, artifact and destination write. Two rules follow, and together they close the gap:
- An exact replay returns the prior result. A speculative duplicate of the same task, with the same identity and the same digest, receives the original outcome rather than starting new paid work.
- The same identity with a different digest is equivocation, and it refuses. A retry that rebuilt the request from moved bytes is rejected instead of quietly charged as a second job.
The response tells you which you received, so log it. Do not infer a replay from timing. See how idempotency and retries work.
Read the output independently of your partitions. result-manifest.json is written last and is the only complete-bundle marker; join accepted rows back by item_id_field, never by file order or partition index.
What is not supported
Lithi makes no wire, SDK or endpoint compatibility claim for any distributed runtime, and does not act as a scheduler for one.
Outside the adapter:
- Running your tasks, actors or executors inside Compute.
- Submitting one request per partition and expecting them to deduplicate. Submit one batch.
- Reading a directory your job is still writing to.
- Treating a duplicate submission with fresh bytes as a safe retry. It refuses.
If a batch is partly complete, the job reports PARTIAL with a next action. Read the cursor, take the accepted parts, and resubmit only the outstanding items.