Shape your output partitions

Where partitions are declared

Your request carries an output object. It declares the format, the destination and the result partitioning for the run.

Declare it before you quote. Partitioning changes the shape of what you receive, so settling it late means reading a bundle you then have to reshape.

Your partitions are yours alone

Result partitions are a formatting choice about the files you receive. They are unrelated to how the work itself was divided internally, which you never manage or see.

They are also independent of how your source was laid out. A source split into a thousand files can return four result files, and a single source file can return many.

So choose partitions for the system that will read them. Nothing upstream constrains the choice, and no downstream reader should have to reverse-engineer it.

Bound four things

Set all four together. Fixing one and leaving the others open is how a well-intentioned layout becomes unreadable.

• File size — pick a target range your reader handles comfortably, and a hard upper bound above it.
• File count — bound the total. A predictable count keeps listing, retries and reconciliation simple.
• Compression — choose a codec your reader supports natively, and apply it consistently across every part.
• Partition cardinality — bound the number of distinct partition values. This is the one that fails worst when left open.

Keep cardinality low

Partition on values with few distinct entries and real query meaning, such as a date or a bounded category. That is what partitioning is for.

Partitioning on a high-cardinality field produces one tiny file per value. Warehouses and object stores both handle that badly, and the cost lands on every read that follows.

If a field has more distinct values than you can name a bound for, do not partition on it. Sort by it inside the file instead.

Sanity-check the arithmetic before you run

Estimate rows, then bytes per row, then total bytes. Divide by your target file size to get an expected file count, and compare it against your bound.

Then multiply your partition values together. If the product is larger than your file-count bound, one of the two is wrong, and it is usually the cardinality.

Doing this before the quote costs far less than reshaping a delivered bundle.

What success looks like

The manifest binds the number of files you expected, at the sizes you asked for, with the compression you declared. Every part sits under a partition value you can name in advance.

Your reader loads them without a reshaping step. A person who has never seen the run can tell which partition holds what, from the path alone.

If the shape comes back wrong

Do not rerun to fix layout. Read the manifest's file list and counts against what you declared, and check for a partition value you did not anticipate in your data.

Where the layout is workable, repartition on your side and keep the delivered bundle as it is. Where it is not, adjust the output declaration for the next run.

If a delivery failed because of file size or count at the destination, that is a delivery problem, not a compute one. Follow retry delivery without rerunning the job, then reconcile against the result bundle layout.