> ## Documentation Index > > Fetch the complete documentation index at: https://lithi.ai/llms.txt > > Use this file to discover all available pages before exploring further. --- title: Normalize and deduplicate a dataset description: >- Clean a small dataset while keeping every item identifier intact. Export accepted and rejected rows apart, declare the schema up front, and check that duplicates never span two partitions. route: /docs/compute/examples/data-preparation page_id: docs-compute-examples-data-preparation page_type: tutorial content_layer: guided documentation surface_profile: guided_docs audience: Data and machine learning teams preparing a dataset for downstream use voice: D reader_question: How do I clean and deduplicate a dataset without losing track of the original rows? primary_action: Run a ten-row preparation test source_locale: en-US source_status: APPROVED source_version: docs-compute-examples-data-preparation-en-us-2026-09-05-v1 last_updated: '2026-09-05' robots: index claim_registry_pinned: true claim_registry_resync: "npm run governed:tsx -- scripts/build-compute-public-claim-registry.ts --write" --- ## What you get A cleaned dataset that you can still trace back to the rows you started with. - Accepted rows in `data/part-*`, normalized against the schema you declared. - Rejected rows in `failures/part-*`, each with a typed code and the reason it failed. - A duplicate decision per group: which identifier survived, and which ones it absorbed. - Counts in `metrics.json` that reconcile against your submitted total. Every one of those carries your original identifier. Normalization changes values; it never renames a row. ## What you need Ten rows and a declared schema. Six of them carry the cases you need to see: ```text record_id,company_name,email_domain,country,created_at REC-001,Northwind Ltd.,northwind.com,US,2026-01-04 REC-002,NORTHWIND LTD,northwind.com,us,2026-02-11 REC-003,Contoso GmbH,contoso.de,DE,2026-01-19 REC-004,Contoso Gmbh,contoso.de,DE,2026-01-19 REC-005,Fabrikam,,GB,2026-03-02 REC-006,,fabrikam.co.uk,GB,not-a-date ``` Declare `record_id` as your `item_id_field`. Rows one and two are the same company written twice, and rows three and four differ only in case. Row five is missing a domain, and row six carries an unparsable date that should be rejected rather than repaired. ## Run it 1. Validate locally: `lithi dataset validate ./prep-fixture/records.csv`. It reports encoding, empty identifiers and duplicate identifiers before anything is sent. 2. Read the workflow: `lithi workflow explain `, and read the parameters that control matching strictness. 3. Declare the target schema explicitly. Name each column, its type and whether it may be null. An implicit schema is how a silent type change reaches production. 4. Quote it: `lithi quote --workflow --input ./prep-fixture/records.csv`. The quote gives a maximum charge and an expiry, and starts nothing. 5. Approve the quote separately, submit with `lithi batch submit`, then track with `lithi status `. Declare your output partitions in the same submission, so accepted and rejected rows land apart rather than in one file you have to filter later. ## Read the result Open `result-manifest.json`, then reconcile the counts before reading any values. Accepted plus rejected plus abstained should equal ten. If it does not, read the job status before assuming anything succeeded. Then run three checks: - **Identity survived.** Every accepted row still carries its `record_id`. A merged pair reports the surviving identifier and the absorbed one, so nothing disappears without a record. - **Rejections are separate and typed.** Row six is in `failures/part-*` with a reason, not silently coerced to a default date. - **Duplicates do not span partitions.** If you split the output for training and evaluation, the same real-world item must not appear on both sides. That leak inflates every downstream result, and it is invisible once the split is made. Check the leak by joining on the deduplication group, not on `record_id`. Two identifiers that were merged are one item, and both sides of the split must agree on that: [how result partitions are declared](/docs/compute/delivery/result-partitioning). ## Where it stops You get a prepared dataset and a rejection list. Nothing is loaded into a warehouse, trained on, or promoted to a production table. Deduplication is a judgment about which rows describe the same thing. The workflow applies the matching rules you declared and reports what it merged. A pair it could not decide abstains rather than merging, and those pairs are yours to review. Keep the mapping from your identifiers back to real records on your own side: [give every item a stable identity](/docs/compute/inputs/item-identity). A person reviews the rejected rows and decides what to fix at the source.