What you’ll learn: How to inspect a sanitized GCP pattern for governed dataset onboarding, auditability, cost visibility, and promotion rules across analytics and ML use cases.
gcpbigquerygovernanceanalyticsml
On this page
Consider a platform design that looks clean until a compliance review exposes gaps.
In this reference scenario, analytics wants faster access to governed BigQuery datasets. ML wants stable feature tables with known lineage. Governance wants policy tags, approval evidence, and auditability. Platform wants a small number of reusable patterns instead of a custom exception for every team.
The trap is sequencing those needs. Build the platform first, add controls later. Let analysts move fast, then make it compliant. Give ML a sandbox, then figure out promotion. Each version of that plan creates the same risk: the system would work by bypassing the rules it was supposed to enforce.
A better design treats compliance as a platform feature.
This is a sanitized reference architecture. Dataset names, volumes, and timings are rounded or illustrative. The pattern exposes architecture tradeoffs; it does not report a named engagement or measured onboarding improvement.
Before state#
Assume a starting point with three lanes, but only one of them is explicit.
- Governed reporting. Slow, reviewed, and trusted once it shipped.
- Ad hoc analytics. Fast, useful, and often hard to trace later.
- ML experimentation. Productive in notebooks, brittle at promotion time.
Shadow paths can emerge when the official path is expensive.
Watch for these symptoms:
- Sensitive columns are documented, but policy enforcement depends on humans remembering the rule.
- Dataset promotion means opening tickets and attaching screenshots.
- Feature tables are copied from analytics models without a clear owner for drift, freshness, or schema change.
- Cost reviews happen after a query pattern is already expensive.
- Access reviews are periodic, not tied to the moment a data product changed.
The constraint is not "be compliant." That is too broad to design against.
The actual constraint is sharper: make the compliant path faster than the workaround.
Design decision#
The platform uses two lanes with shared controls:
| Lane | Purpose | Promotion rule |
|---|---|---|
| Experimental | Exploration, notebook work, prototype features | Time-limited access, no executive reporting, no production ML dependency |
| Governed | Reporting, published marts, ML feature tables | Tests, policy tags, owner, release approval, cost guardrail |
The important part is that both lanes use the same primitives: BigQuery datasets, service accounts, IAM groups, Dataform release workflows, policy tags, and platform telemetry. The promotion contract distinguishes the lanes.
Diagram source (text)
flowchart LR Raw["Raw landing dataset"] --> Curated["Curated governed dataset"] Curated --> Reporting["Reporting marts"] Curated --> Features["ML feature tables"] Sandbox["Experimental dataset"] --> Review["Promotion review"] Review --> Curated Policy["Policy tags and grants"] --> Curated Tests["Dataform tests"] --> Review Cost["Query cost telemetry"] --> Review Owner["Named data owner"] --> Review
The tradeoff#
The tradeoff is accepting stricter promotion in exchange for faster approved reuse.
One alternative is to let teams copy data into their own datasets and ask governance to review later. That is faster for the first request and slower for every request after it. It multiplies policy surfaces, duplicates definitions, and makes access review harder.
Another alternative is to force every exploratory question through the governed lane. That protects the platform, but it destroys learning speed.
The two-lane design gives both sides a place:
- experiments can happen without pretending to be production,
- production assets have explicit ownership and tests,
- ML feature tables do not get special exemptions,
- access reviews attach to data products instead of one-off tickets.
Operating artifact: promotion checklist#
The promotion checklist is deliberately short enough to use in a pull request.
| Check | Required evidence |
|---|---|
| Owner | Named data product owner and Slack/escalation route |
| Classification | Policy tags on sensitive columns and documented data class |
| Contract | Grain, primary keys, null expectations, and accepted value ranges |
| Tests | Dataform assertions for breaking schema and quality changes |
| Access | IAM group or service account reviewed for least privilege |
| Cost | Expected query pattern and monthly cost risk noted |
| Freshness | SLA and stale-data behavior documented |
| Downstream use | Reporting, ML, or operational consumers listed |
| Rollback | Last known good release or disable path available |
This checklist makes governance executable within the release path.
Intended outcomes and measures#
Proposed goals for an implementation are:
- reduce governed dataset onboarding time for standard patterns against a recorded baseline,
- remove recurring manual review for low-risk schema changes that passed policy and test gates,
- make feature-table promotion use the same path as reporting marts,
- expose query cost by data product owner,
- give auditors evidence from the release path instead of reconstructing it from tickets.
Measure the time from "dataset is ready for promotion" to "approved governed asset," comparing equivalent request types before and after implementation. Supplying contracts, tests, ownership, policy tags, cost expectations, and consumer lists is intended to reduce approval uncertainty; this article does not report a measured reduction.
Track unclear access tickets, promotion failures, approval lead times, and cost attribution coverage. Treat reduced shadow-data risk as a hypothesis until supporting observations are available.
What I would change next time#
I would add cost guardrails earlier.
Security and access usually get the first governance attention, but runaway query cost is also a governance problem. A table that is technically compliant but financially invisible can still damage trust in the platform.
The next version would make these fields part of the first promotion request:
| Field | Example |
|---|---|
| Expected consumer | Weekly executive dashboard, daily feature pipeline |
| Query pattern | Scheduled aggregate, analyst exploration, batch scoring |
| Cost owner | Platform, product analytics, ML team |
| Alert threshold | Job cost or scanned bytes above agreed limit |
| Remediation | Partitioning, materialized table, semantic-layer change, or access review |
Compliance platforms work when the safe path is also the easy path. On GCP, that means using BigQuery, IAM, policy tags, Dataform, and telemetry as one operating system, not as disconnected controls.