Dataform + BigQuery Governance Release Patterns

What you’ll learn: How to inspect a sanitized release pattern that makes schema, freshness, cost, and downstream-impact checks part of promotion rather than after-the-fact review.

Consider an illustrative near miss that looks harmless.

A proposed transformation change renames a field that only one downstream dashboard is expected to use. The model builds successfully and looks correct in development. Local tests pass, making the pull request appear safe.

In this scenario, the dashboard is not the only consumer. A scheduled export and an ML feature job also depend on the same field. The break could reach production if promotion checks only the changed model.

This scenario illustrates how I approach Dataform release patterns on BigQuery. The release unit is not the SQL file. It is the downstream contract.

This is a sanitized reference architecture. Table names, ownership names, and incident counts are illustrative. It documents a release shape I use; it does not report a named engagement or a measured customer outcome.

Before state#

Assume a platform with reasonable tools and weak release boundaries.

Dataform compiles transformations.
BigQuery stores raw, curated, and reporting tables.
Analysts review SQL changes.
Governance reviews sensitive data handling.
Downstream teams learn about breaking changes through Slack or broken jobs.

The missing piece is not testing in general. It is testing the contract that other people depended on.

The constraint is delivery speed. The answer cannot be "make every change wait for a committee." The release path needs to catch dangerous changes without making routine modeling work unbearable.

Release lanes#

I separate Dataform changes into three lanes.

Lane	Examples	Gate
Safe internal change	Refactor CTE, add unused column, improve naming inside a private model	Compile, unit assertions, owner approval
Contract change	Rename/remove column, change grain, change semantics, alter freshness expectation	Downstream impact, contract approval, migration plan
Cost-risk change	New join path, larger scan, incremental strategy change, backfill	Cost estimate, dry-run bytes, rollback plan

The lane matters because the approval question changes. A contract change is not risky because SQL is hard. It is risky because someone else built a decision on top of the old shape.

Promotion checks the changed model, expected cost, affected contracts, and owner approval before production.

Diagram source (text)


sequenceDiagram
  participant Dev as Developer
  participant DF as Dataform
  participant BQ as BigQuery
  participant Gate as Release gate
  participant Owner as Data owner
  participant Prod as Production

  Dev->>DF: Open change
  DF->>DF: Compile and run assertions
  DF->>BQ: Dry-run changed models
  BQ-->>Gate: Bytes scanned and affected tables
  Gate->>Gate: Classify safe, contract, or cost-risk
  alt Contract or cost-risk
    Gate->>Owner: Request approval with migration plan
    Owner-->>Gate: Approve or block
  end
  Gate->>Prod: Promote release
  Prod-->>Gate: Smoke check freshness and row counts

The tradeoff#

This pattern adds friction to contract and cost-risk changes.

The gate exists because the friction belongs where the blast radius is real.

One alternative is a blanket approval process for every model change. That protects the platform by making everyone slower, which eventually pushes teams back into side channels. Another alternative is pure autonomy: let every team own its own models and depend on conventions. That keeps velocity high until a silent contract break makes trust expensive.

The middle path is lane-based release discipline.

Safe internal changes stay fast. Contract changes get explicit. Cost-risk changes get measured before production.

Contract artifact#

For governed models, the contract belongs beside the model, not in a wiki.

orders_daily.contract.yaml

model: orders_daily
owner: revenue-analytics
grain: one row per order date and market
primary_key:
  - order_date
  - market_id
freshness_sla: "daily by 08:00 ET"
breaking_change_requires:
  - owner_approval
  - migration_note
  - downstream_consumer_notification
columns:
  gross_revenue:
    type: numeric
    nullable: false
    semantic: pre-refund gross booked revenue
  net_revenue:
    type: numeric
    nullable: false
    semantic: gross revenue minus refunds and credits

The example is illustrative, but the fields are practical. Grain, keys, nullability, freshness, and semantic meaning are the places where "the dashboard looks wrong" usually starts.

Validation gates#

The release gate checks four things before promotion.

Gate	Signal	Why it matters
Compile	Dataform compile succeeds	Catches syntax and dependency graph issues
Contract	Grain, keys, required columns, type expectations	Catches breaking downstream changes
Cost	BigQuery dry-run bytes or changed scan pattern	Catches expensive changes before they surprise finance
Smoke	Row count, freshness, null spike, downstream job health	Catches regressions immediately after promotion

The smoke check is intentionally boring. Row counts, freshness, and null spikes catch more real incidents than elegant checks that nobody maintains.

Rollback behavior#

Rollback should be straightforward too.

For governed models, choose and document a rollback plan before release:

revert the Dataform release and republish,
restore a table snapshot,
disable a downstream schedule,
publish a compatibility column while consumers migrate,
freeze a backfill until a cost-risk change is reviewed.

The important decision is made before release: which rollback path is available for this model?

If the answer is "we will figure it out when it breaks," the model is not ready for governed promotion.

Intended outcomes and measures#

The intended result is fewer surprise breaks reaching consumers. This reference pattern does not establish that result. Evaluate an implementation using:

contract-break incidents per month,
percentage of governed models with explicit owners,
promotion lead time by lane,
BigQuery dry-run bytes for cost-risk changes,
recovery time after a failed smoke check.

These measures make release risk observable; collect a baseline before judging improvement.

One proposed target is: safe internal changes should stay same-day, while contract changes should include a migration plan before production. If those two lanes take the same amount of time, the process is probably too heavy for one of them and too light for the other.

What I would change next time#

I would make downstream registration explicit earlier.

Inferring consumers from known dashboards, scheduled queries, and feature jobs can miss informal dependencies. A consumers.yaml file per governed model would make those dependencies and their owners explicit:

orders_daily.consumers.yaml

model: orders_daily
consumers:
  - name: executive-revenue-dashboard
    owner: revenue-analytics
    dependency: dashboard
  - name: churn-feature-batch
    owner: ml-platform
    dependency: feature_table
  - name: finance-close-export
    owner: finance-ops
    dependency: scheduled_export

Data contracts work when they are treated as release artifacts. Dataform and BigQuery are strong enough to support that pattern, but the tools do not create the discipline on their own. The discipline is deciding which changes are safe, which changes are contracts, and which changes need a rollback plan before they touch production.