Prepare data with pandas, Polars or Arrow

Keep dataframe preparation local

Your application selects the columns and types; your agent keeps the task goal, context, planning and final decision. The selected client or Kit helper carries the accepted file representation when that workflow provides one. Lithi receives the approved artifact reference and returns a result that you join and check locally.

Use your existing Python environment to select authorized columns and validate a small typed dataset. The examples below create synthetic bytes locally. They do not call Compute or establish a native dataframe connector. Use the listed libraries from your environment's reviewed dependency set.

Keep a stable string item ID before filtering or sorting. Do not use a dataframe index as identity unless your application explicitly guarantees its stability. Agree how the workflow represents nulls, timestamps, decimals, binary values and nested data before you export.

The examples use an explicit item_id string and nullable text value. Their round-trips show the result you need to preserve: record identity stays stable, and a missing value remains JSON null rather than becoming text.

Write JSONL with pandas

This self-contained example requires pandas. It preserves a null rather than turning it into the string None.

import json
import pandas as pd
rows = [{"item_id": "sample-001", "text": "Hello"},
        {"item_id": "sample-002", "text": None}]
frame = pd.DataFrame(rows, columns=["item_id", "text"])
if frame["item_id"].isna().any() or not frame["item_id"].is_unique:
    raise ValueError("Missing or duplicate item IDs")
payload = frame.to_json(orient="records", lines=True, force_ascii=False)
assert [json.loads(line) for line in payload.splitlines()] == rows
print(payload, end="")

The expected output contains two JSON objects on separate lines. The second object's text is JSON null. See pandas' to_json reference for format and date options; don't rely on defaults for business timestamps.

Use Polars for the same local records

This independent example requires Polars and produces newline-delimited JSON using its write_ndjson method.

import json
import polars as pl
rows = [{"item_id": "sample-001", "text": "Hello"},
        {"item_id": "sample-002", "text": None}]
payload = pl.DataFrame(rows).write_ndjson()
assert [json.loads(line) for line in payload.splitlines()] == rows
print(payload, end="")

Round-trip Parquet with Arrow

This independent example requires PyArrow. It writes only an in-memory buffer, then checks the decoded records. Arrow's write_table and read_table references document the methods used below.

import pyarrow as pa
import pyarrow.parquet as pq
rows = [{"item_id": "sample-001", "text": "Hello"},
        {"item_id": "sample-002", "text": None}]
sink = pa.BufferOutputStream()
pq.write_table(pa.Table.from_pylist(rows), sink)
encoded = sink.getvalue()
restored = pq.read_table(pa.BufferReader(encoded)).to_pylist()
assert restored == rows
print({"rows": len(restored), "bytes": len(encoded)})

The byte count depends on your encoder version. Record that version and freeze the actual exported bytes; different encoders need not produce identical files. This local round-trip doesn't establish that a Compute workflow accepts Parquet.

Hand off the approved representation

Map the task brief to the selected workflow and its output contract before handing off the data. Check the selected workflow's accepted format and field schema. Keep the item mapping, exact file bytes and digests in your preparation record. Keep storage credentials in your approved local configuration. Use the encrypted artifact workflow rather than inserting a dataframe, local path or storage credential into a native job request.

Review current access and the quote before approving paid work. Lithi processes the admitted artifact; local encoding isn't submission or execution. For large datasets, use bounded reads and exports appropriate to your environment rather than collecting the whole dataset into one process.

Check results and recover

Verify the accepted result before loading it into a dataframe. Join by item ID, review exceptions and reject unexpected or duplicate IDs. If parsing changes a value or type, repair the mapping before further work. If result retrieval or local import fails, reuse the existing job and result instead of resubmitting inference.

Next, prepare the encrypted artifact after the local representation checks pass.