Prepare tabular data for Compute

Before you start

You need the released format profile for the format you are writing, the workflow's generated parameters_schema, and a column that identifies each row.

The input object names the format you chose. That profile decides how a value is read — not your writer's defaults, and not the reader you happen to use locally.

Say what a missing value means

Most tabular defects are null defects, and they never raise an error.

Your data may use a true null, an empty string, or a sentinel such as N/A, -1 or unknown. Pick one and use only that one. A sentinel is a real value, so a workflow reads -1 as a number and grades it.

Convert sentinels to real nulls before you submit. Where a blank genuinely means something different from missing, keep both and say so in the schema, rather than hoping the distinction survives.

Pin the logical type, and let unknown fields refuse

A column's storage type is rarely the whole answer. State the logical type your data actually holds:

Round-tripping is the fast test. Write the file, read it back with a different reader, and compare. Anything that changes on the way back will change again in a run.

Nested fields follow the same rule. Read the parameters_schema before you nest anything, because a struct where the workflow declared a flat field is not converted for you. An unknown or unsupported field is refused rather than dropped or coerced, and the refusal names it. A silently coerced field would change the work you are paying for and still look successful.

Write timestamps that carry their own offset

Naive local time is the most expensive small mistake in this list. A timestamp with no offset is read against whatever the reader assumes, and a whole column can shift by hours without a single warning.

Write an explicit offset, or write UTC and say so. Use ISO 8601 form, keep one precision across the column, and keep dates and timestamps in separate columns rather than mixing them.

If your source is genuinely local wall-clock time, add the offset column beside it. Do not convert and discard the original.

Validate a deliberately awkward sample

Build a small sample that contains your edge cases on purpose, then run lithi dataset validate against it locally.

Include one row for each of these:

• a null of every kind you have
• the longest string in the table
• the earliest and the latest timestamp
• an unusual category value, and a non-ASCII value
• an empty nested list
• an identifier that looks confusingly like another one

Check the identifiers too. Each row needs a stable value in the field you declared, present everywhere and repeated nowhere: see how item identity works.

What success looks like

Validation passes on the awkward sample, not just on the easy one. Your quote echoes the resolved parameters, the maximum charge and the maximum item count that applies to the workflow.

Read those before approving. A quote never authorizes work, so nothing runs until you approve it separately.

If a row or a column is rejected

The refusal names the field and the row. Fix it at the source rather than patching the file you already pinned.

Correcting the data changes the bytes, so take a new snapshot and quote again with the new digest.

If the format profile itself has moved, admission answers REPLAN_REQUIRED. Re-read the released format profiles at the version your quote names, then rebuild from there.