Dataform vs. script piles: how we keep transformations reviewable

A script is cheap while it’s local, temporary, and owned by the person running it. Once other systems depend on its output, the script has become a data product without admitting it.

That’s when we move transformation logic into Dataform.

The deciding variable isn’t file extension. It’s whether the work now needs a shared dependency graph, repeatable environments, reviewable materialization, assertions, and one production path.

Script piles grow through exceptions

The first script usually does one clear job. The pile forms around it.

A second script handles yesterday’s correction. A scheduler passes a flag that changes the date window. A helper deletes partitions before a rebuild. A backfill copies most of the production logic but uses a different entry point. The table still appears at the end, so the arrangement survives.

Review becomes reconstruction. To understand one output, a reviewer has to trace shell commands, Python helpers, SQL files, scheduler arguments, temporary tables, and whatever order those pieces happen to run in. The system’s behavior is procedural history rather than a visible model.

The code may be short. The explanation required to change it isn’t.

A modeling layer gives the output a home

In Dataform, a production table is a named action with configuration, dependencies, SQL, assertions, and documentation close enough to review together.

Use ref() or declared dependencies so the graph represents the data contract rather than relying on filenames and scheduler order. Keep materialization settings with the model. Put incremental keys, partition filters, and pre- or post-operations where a reviewer can see that they change how the table is built.

Assertions belong beside the claim they protect. If an output is unique by order_id, the repository should make that visible and testable. If a dependency must not contain null customer identifiers, the assertion should fail before downstream reports quietly accommodate them.

The benefit isn’t that Dataform makes SQL correct. It makes the system’s intended build graph inspectable.

Compile before production owns the surprise

A transformation review should include more than the source .sqlx diff.

Inspect the compiled SQL for meaningful changes, especially when JavaScript, macros, variables, or incremental branches are involved. Review the dependency graph and the set of actions selected for execution. Estimate affected partitions and bytes before a large rebuild. Run assertions against an isolated schema or controlled development environment.

The repository should make it cheap to answer:

• which models will rebuild
• which downstream objects depend on this change
• whether production and development compile the same logic
• what incremental branch will execute
• what assertions can fail
• how to run the same model for a backfill

A platform that can answer those questions in code review has removed a large amount of incident archaeology.

One model should have one production path

Backfills, scheduled runs, and manual repair should invoke the same model logic with explicit parameters or bounded operations. They shouldn’t each maintain a slightly different script.

A backfill may choose a wider date range or force a full refresh. That’s operational variation around one model. If the backfill changes business filters, joins, or output grain, it’s a second implementation and will eventually disagree with the first.

Keep the variation visible. A model that behaves differently under an undocumented workflow flag isn’t declarative in any useful sense.

Dataform isn’t an excuse to hide code inside SQLX

JavaScript and reusable helpers can remove mechanical repetition. They can also make every model depend on an abstraction that has to be mentally executed before the SQL makes sense.

Use code for stable generation, shared declarations, metadata, or logic that’s genuinely clearer outside SQL. Keep business grain, joins, filters, and measures visible in the model. A reviewer shouldn’t have to open five helpers to learn what qualifies an active customer.

The boundary is covered in How we decide whether a transformation belongs in SQLX, code, or orchestration. The practical test is whether the abstraction lowers review cost for the caller.

Orchestration stays outside the semantic layer

A workflow may invoke Dataform, select tags, enforce concurrency, retry infrastructure failures, and coordinate external ingestion. It shouldn’t decide what the table means.

Dependencies that Dataform can express should live in the graph. Cleanup required for a model’s correctness should live with the model or in a named operation it owns. Business conditions shouldn’t be encoded as scheduler branches.

This keeps orchestration thin and gives model review one center of gravity.

When a script is still the right tool

Use a script for exploratory analysis, a one-time migration, local data inspection, or an external interaction that doesn’t define a shared analytical table. Scripts are also useful at system boundaries where API calls, files, or non-relational algorithms are the actual work.

Promote the script when it becomes scheduled, shared, retried, incremental, monitored, or depended on by another model. At that point the platform is already paying production cost. Refusing to give the logic a production structure only hides the bill.

We choose Dataform once transformation behavior must survive shared change. Named models, explicit dependencies, compiled output, assertions, and one execution path make that change reviewable. The tool earns its place by replacing memory with structure, not by moving the same script pile into a more respectable directory.