Dataform assertions: make failures block downstream models

A Dataform run can report a failed uniqueness assertion while a downstream table has already built from the duplicate rows. Depending on the table is not the same as depending on its assertions. Google’s dependency documentation makes that distinction explicit.

For data that must be checked before it reaches reports or automation, we’d build a separate candidate, validate it, and make publication depend on the checks. The serving table should remain on its last accepted version when the candidate fails.

Build a candidate that readers don’t use

The following is a small example for an initialized Dataform project targeting BigQuery. It assumes raw.orders already exists in the configured project and contains order_id, customer_id, and net_amount. Use isolated development datasets to try it; the candidate and mart datasets need to be writable by the execution identity.

First, declare the existing source in definitions/raw_orders.sqlx:

config {
  type: "declaration",
  schema: "raw",
  name: "orders",
}

Then create definitions/orders_candidate.sqlx:

config {
  type: "table",
  schema: "candidate",
  assertions: {
    nonNull: ["order_id"],
    uniqueKey: ["order_id"],
  },
}

select
  order_id,
  customer_id,
  net_amount,

from
  ${ref("raw", "orders")}

The candidate deliberately doesn’t hide duplicates with distinct or choose an arbitrary row. Its intended grain is one row per order, so a duplicate should fail the check until the model has an explicit rule for resolving it. Google’s Dataform assertion documentation explains that built-in assertions run after the table is created; they don’t roll back its creation.

Keep reports and operational consumers on the serving dataset. Where appropriate, deny those identities access to the candidate dataset as well. Otherwise someone can bypass the publication step by pointing a report at the convenient intermediate table.

Make publication wait for the candidate’s assertions

Create definitions/orders_published.sqlx:

config {
  type: "table",
  schema: "mart",
}

select
  order_id,
  customer_id,
  net_amount,

from
  ${ref({
    schema: "candidate",
    name: "orders_candidate",
    includeDependentAssertions: true,
  })}

The object passed to ref adds the candidate’s direct assertions as dependencies of this action. The destination is mart.orders_published. When a required assertion fails in the invocation, Dataform doesn’t execute the publication action.

Use a materialized serving table here, not an ordinary view over the mutable candidate. A view that already exists can expose changed candidate rows even when a later attempt to recreate the view is blocked. Similarly, checking the serving table only after overwriting it won’t preserve the previous valid result.

For an action that must wait for the direct assertions of all its dependencies, Dataform also supports dependOnDependencyAssertions: true in its configuration. Per-dependency includeDependentAssertions settings take precedence. These options refer to direct assertions; don’t assume they recursively attach every check anywhere upstream. Inspect the compiled graph at the publication points you care about.

Add business checks explicitly

Non-null and unique keys won’t establish that a load contains all required orders. A reconciliation check might compare the candidate with an export manifest, while a relationship check might reject orders with missing customers.

Write each check as a named assertion that returns violating rows, using the same run or source boundary as the candidate. Then make the publisher depend on that named assertion. For example, a separately implemented orders_reconciled assertion can be required through:

config {
  type: "table",
  schema: "mart",
  dependencies: ["orders_reconciled"],
}

This is an addition to the publisher’s configuration, not a second complete model. Keep the candidate ref with includeDependentAssertions: true in its query. The named assertion must exist and depend on the candidate or its relevant source evidence, never on the table it is supposed to protect. That would create a dependency cycle.

Choose blocking checks based on whether their failure makes publication unsafe. A completeness failure may need to block a financial report; a nonessential description field may warrant a warning without stopping the entire warehouse.

The selected invocation must include the checks

The dependency graph describes ordering, but a selective run still needs to include its prerequisites. In the API, an invocation configuration selecting this publisher could use:

{
  "includedTargets": [
    {
      "database": "YOUR_PROJECT_ID",
      "schema": "mart",
      "name": "orders_published"
    }
  ],
  "transitiveDependenciesIncluded": true
}

Use the target identifiers from the actual compilation result, including any environment suffixes or overrides. Google’s Dataform InvocationConfig reference defines selection and dependency inclusion. This object is the invocation configuration, not the entire workflow-invocation request.

Including dependencies may rebuild more upstream work than a manually selected publisher alone. Test the scheduled configuration and the operational rerun path, not only the broad “run everything” action used during development.

Test failure and recovery, including overlapping runs

Start with a valid published result. Introduce a duplicate order in the test source and run the same selected invocation used in production. Confirm that the candidate changes, the uniqueness assertion fails, the publication action doesn’t run, and the serving table retains its previous contents. Repeat with a null key, then repair the input and verify that a subsequent run publishes the corrected result.

This example uses a fixed candidate table name, so overlapping runs need serialization. Otherwise one invocation can validate a candidate that another invocation replaces before publication. For concurrent processing, use run-isolated candidates and carry the release identity through checks and publication.

A successful gate also doesn’t make several serving-table updates one transaction. BigQuery has transaction restrictions documented by Google, including on permanent-table DDL. Use an explicit release design when consumers require several tables to change together.

Show the last successful publication time. Freshness monitoring should identify when consumers are using an older result and stop time-sensitive consumers when that result becomes too old.