What we keep out of orchestration in data platforms

Orchestration should decide when work runs, not what the resulting data means.

Once a workflow owns business filters, deduplication, cleanup semantics, or model grain, the platform has two semantic layers: the model people review and the scheduler that secretly changes it. That arrangement remains convenient only for the person who remembers every branch.

What orchestration should own

A workflow is the right place for operational coordination:

• schedules and event triggers
• cross-system dependencies
• retries, timeouts, and failure notification
• concurrency and mutual exclusion
• runtime identity and credentials
• selecting a bounded backfill range
• invoking external APIs or file transfers
• recording execution state

These decisions describe how work enters and moves through the platform. They should be explicit enough that an operator can tell what ran, what is waiting, and what can be retried.

Business semantics stay in models

Keep grain, joins, filters, classifications, measures, key logic, and source precedence in the transformation layer.

The scheduled run and a manual backfill should invoke the same model definition. The backfill may widen dates or affected keys. It shouldn’t use a second SQL file with slightly different business logic because “backfills are special.” That’s how old periods acquire their own private definition of revenue.

When one schedule legitimately needs another semantic output, name another model. Don’t hide the distinction behind a runtime flag.

Cleanup belongs with the state it repairs

Deletes, partition replacement, stale-row correction, and merge behavior belong close to the model whose correctness depends on them.

Orchestration can serialize the operation or pass the affected partitions. It shouldn’t contain the only implementation of how stale rows are removed. A reviewer inspecting the model must be able to see its correction and deletion contract without opening a workflow graph.

Avoid generic “cleanup” tasks that mutate several tables after the declarative graph finishes. They create a period where published data is known to be wrong and make retries difficult to reason about.

Don’t use workflow branches as a rule engine

Conditions such as “run this table only for premium customers,” “exclude Fridays,” or “use another metric after month end” are business behavior. Put them in reviewed data or configuration owned by the relevant product, not task branching.

Workflow conditions should be operational: source file exists, upstream execution succeeded, manual approval was granted, or a deployment window is open.

The distinction is whether the branch changes the meaning of data or only whether an already-defined operation may run.

Keep retry behavior idempotent

A workflow retry should repeat the same logical action against the same input boundary. It shouldn’t switch SQL, widen the date range, skip validation, or invoke a “repair mode” that produces a different result.

If retry requires another behavior, model that behavior as a named recovery action with its own contract and audit. Hidden fallback paths are pleasant during the first incident and impossible to trust during the fifth.

Side effects need stable idempotency keys or execution identifiers. Orchestration can coordinate them, but the target operation still owns how duplicate attempts converge.

Let the model graph express model dependencies

If Dataform or another declarative tool can infer and execute the dependency, don’t duplicate that graph manually in the workflow.

The orchestrator can trigger a selected set of tags or actions and wait for completion. Hand-maintained task-per-table DAGs create two sources of truth and make every model dependency change an orchestration change as well.

Use the workflow for dependencies outside the modeling system: extraction finished, transformation graph completed, export delivered, downstream API called.

Make parameters boring

Good orchestration parameters describe execution boundaries: environment, start date, end date, tenant, source batch, or full-refresh intent.

Bad parameters alter hidden semantics through Boolean switches such as legacy_mode, skip_bad_customers, or use_new_logic. Those flags create combinatorial programs few reviewers test.

When semantics change, version the model or configuration and deploy it through normal review. Don’t ask a scheduler invocation to choose which truth applies tonight.

Keep the graph readable under failure

An operator should be able to tell which external dependency failed, which transformation graph ran, whether a retry is safe, and which side effects may already have happened.

If the graph contains dozens of tiny tasks only to expose every SQL statement, it may be over-modeled. If it contains one opaque task that performs ingestion, transformation, cleanup, and export, it’s under-modeled. Boundaries should follow independently recoverable work.

Keep the split explicit

Use orchestration for triggers, coordination, bounded parameters, retries, and external handoffs. Keep analytical meaning and state correction in the model layer. Keep non-relational algorithms in code with explicit data contracts.

Thin orchestration prevents the scheduler from becoming an undocumented second application whose behavior changes alongside metrics without the same review.