Compute
Normalize and deduplicate a dataset
Clean a small dataset while keeping every item identifier intact. Export accepted and rejected rows apart, declare the schema up front, and check that duplicates never span two partitions.
What you get
A cleaned dataset that you can still trace back to the rows you started with.
- Accepted rows in
data/part-*, normalized against the schema you declared. - Rejected rows in
failures/part-*, each with a typed code and the reason it failed. - A duplicate decision per group: which identifier survived, and which ones it absorbed.
- Counts in
metrics.jsonthat reconcile against your submitted total.
Every one of those carries your original identifier. Normalization changes values; it never renames a row.
What you need
Ten rows and a declared schema. Six of them carry the cases you need to see:
record_id,company_name,email_domain,country,created_at
REC-001,Northwind Ltd.,northwind.com,US,2026-01-04
REC-002,NORTHWIND LTD,northwind.com,us,2026-02-11
REC-003,Contoso GmbH,contoso.de,DE,2026-01-19
REC-004,Contoso Gmbh,contoso.de,DE,2026-01-19
REC-005,Fabrikam,,GB,2026-03-02
REC-006,,fabrikam.co.uk,GB,not-a-date
Declare record_id as your item_id_field. Rows one and two are the same company written twice, and rows three and four differ only in case. Row five is missing a domain, and row six carries an unparsable date that should be rejected rather than repaired.
Run it
- Validate locally:
lithi dataset validate ./prep-fixture/records.csv. It reports encoding, empty identifiers and duplicate identifiers before anything is sent. - Read the workflow:
lithi workflow explain <workflow-id>, and read the parameters that control matching strictness. - Declare the target schema explicitly. Name each column, its type and whether it may be null. An implicit schema is how a silent type change reaches production.
- Quote it:
lithi quote --workflow <workflow-id> --input ./prep-fixture/records.csv. The quote gives a maximum charge and an expiry, and starts nothing. - Approve the quote separately, submit with
lithi batch submit, then track withlithi status <job-id>.
Declare your output partitions in the same submission, so accepted and rejected rows land apart rather than in one file you have to filter later.
Read the result
Open result-manifest.json, then reconcile the counts before reading any values.
Accepted plus rejected plus abstained should equal ten. If it does not, read the job status before assuming anything succeeded.
Then run three checks:
- Identity survived. Every accepted row still carries its
record_id. A merged pair reports the surviving identifier and the absorbed one, so nothing disappears without a record. - Rejections are separate and typed. Row six is in
failures/part-*with a reason, not silently coerced to a default date. - Duplicates do not span partitions. If you split the output for training and evaluation, the same real-world item must not appear on both sides. That leak inflates every downstream result, and it is invisible once the split is made.
Check the leak by joining on the deduplication group, not on record_id. Two identifiers that were merged are one item, and both sides of the split must agree on that: how result partitions are declared.
Where it stops
You get a prepared dataset and a rejection list. Nothing is loaded into a warehouse, trained on, or promoted to a production table.
Deduplication is a judgment about which rows describe the same thing. The workflow applies the matching rules you declared and reports what it merged. A pair it could not decide abstains rather than merging, and those pairs are yours to review.
Keep the mapping from your identifiers back to real records on your own side: give every item a stable identity. A person reviews the rejected rows and decides what to fix at the source.