Agent Repo Trust Gates: Conftest Policies, SLSA Provenance, and SBOM in GitHub Actions

A skill manifest shipped in a PR that also added a legitimate new retrieval feature. The Python was reviewed; the manifest was skimmed. Nobody caught that tools: [execute_shell] had been added to the skill's capability block. The agent went to production with shell access nobody authorized, and the first sign of it was an unexpected filesystem write three days later.

Standard code review is not built to catch this. Reviewers look for logic errors, not for capability declarations buried in YAML. The policy gate that catches capability escalation has to run before the merge, and it has to be machine-enforced — a human skimming a three-line diff inside a 400-line PR is not the right check.

This post wires two GitHub Actions workflows and a Rego policy that work correctly, and fixes three mistakes that appear in almost every "drop-in starter pack" circulating online. The mistakes are not cosmetic. Two of them produce silent failures — gates that appear to run but don't — and one of them references a CLI binary that does not exist.

The trust surface for AI repos is different#

Standard CI validates code correctness. It does not validate capability scope, build provenance, or dependency inventory. For an AI agent repo, three artifact classes carry production risk that linting and testing don't touch:

Capability manifests. skill.yaml, AGENTS.md, tool-call schemas. These files declare what the agent is allowed to do — not how the code implements it. A one-line change to permissions.network: true or tools: [execute_shell] is a capability decision. It gets the same review weight as a comment change unless the policy gate is there to flag it.

Infrastructure configuration. Kubernetes manifests, IAM policy files, Helm charts. Syntax linters confirm the YAML is valid. They do not confirm whether a new service account was granted roles/storage.admin when it only needed roles/storage.objectViewer. The over-permissioned account ships because the YAML was syntactically correct.

Built OCI images. After the code merges, what built the image? Which runner? Which dependency versions resolved at build time? An image without a provenance record is a claim with no receipts. You can verify the code at the commit; you cannot verify the build environment without attestations.

The gate structure maps to these three classes: conftest policy evaluation for manifests, SLSA provenance for images, and an SBOM for dependency inventory. Each one is cheap to add when the build is clean; expensive to retrofit after an incident.

See also: Do not let an awesome-list become a supply chain for agent behavior and artifact trust boundaries in sandboxed agent pipelines.

Pre-merge policy gate — what actually works#

conftest runs on structured data, not Markdown#

Conftest evaluates OPA/Rego policies against structured input files. The formats it supports natively: JSON, YAML, TOML, HCL, Dockerfile, XML, CUE, SPDX, and several others. Markdown is not on the list.

The line conftest test AGENTS.md --policy policy/agents does not work. Conftest cannot parse a .md file. It will error with an unsupported format message, and the CI step will fail before any policy logic runs. If the step is set to continue-on-error: true (common in starter-pack templates), it fails silently and looks green.

The fix is to target the files that are already structured. Skill manifests are YAML. Infrastructure configs are YAML or JSON. Those are the right conftest inputs.

For AGENTS.md content validation — if you need to enforce that required sections exist — a shell script is the practical tool:

required=("## Objectives" "## Tools" "## Guardrails" "## Evaluation")
for section in "${required[@]}"; do
  grep -qF "$section" AGENTS.md || { echo "Missing section: $section"; exit 1; }
done

This is less expressive than Rego, but it runs without a parser. If the section structure matters enough to gate on, it matters enough to put in YAML and validate it with conftest.

Rego policy for skill manifests#

A minimal policy that catches the execute_shell class of incident and enforces that permissions are explicitly declared:

# policy/skills/skill.rego
package skills
 
required_keys := {"name", "description", "tools", "permissions"}
 
deny[msg] {
    some key
    required_keys[key]
    not input[key]
    msg := sprintf("skill.yaml missing required key: %s", [key])
}
 
banned_tools := {"execute_shell", "run_bash", "shell", "subprocess"}
 
deny[msg] {
    tool := input.tools[_]
    banned_tools[tool]
    msg := sprintf("Banned tool: %s. Requires an explicit policy exception.", [tool])
}
 
deny[msg] {
    input.permissions.network == true
    not input.permissions.network_justification
    msg := "network: true requires a network_justification field"
}
 
deny[msg] {
    input.permissions.filesystem == "write"
    not input.permissions.filesystem_paths
    msg := "filesystem write requires an explicit filesystem_paths allowlist"
}

A skill.yaml that would pass:

name: document-search
description: Retrieves relevant documents from the vector store given a query.
tools: [vector_search, rerank]
permissions:
  network: false
  filesystem: read

A skill.yaml that would be blocked by three rules simultaneously:

name: exec-helper
tools: [execute_shell]          # banned tool
permissions:
  network: true                 # no network_justification
  filesystem: write             # no filesystem_paths

The deny messages surface in the CI log next to the PR, before the merge happens.

Corrected pre-merge workflow#

# .github/workflows/pre-merge-trust-gate.yml
name: pre-merge-trust-gate
on:
  pull_request:
    types: [opened, synchronize, reopened]
    branches: [main]
 
permissions:
  contents: read
 
jobs:
  policy-gate:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
 
      - name: Install conftest
        run: |
          curl -L https://github.com/open-policy-agent/conftest/releases/latest/download/conftest_Linux_x86_64.tar.gz \
            | tar xz && sudo mv conftest /usr/local/bin/
 
      - name: Validate skill manifests
        run: conftest test skills/**/skill.yaml --policy policy/skills
 
      - name: Validate infrastructure manifests
        run: conftest test infra/**/*.yaml infra/**/*.json --policy policy/infra
 
      - name: Check AGENTS.md required sections
        run: |
          required=("## Objectives" "## Tools" "## Guardrails" "## Evaluation")
          for s in "${required[@]}"; do
            grep -qF "$s" AGENTS.md || { echo "Missing: $s"; exit 1; }
          done
 
  attestation-verify:
    runs-on: ubuntu-22.04
    if: hashFiles('images.txt') != ''
    steps:
      - uses: actions/checkout@v4
 
      - name: Install cosign
        run: |
          curl -L https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 \
            -o cosign && chmod +x cosign && sudo mv cosign /usr/local/bin/
 
      - name: Verify attestations for referenced images
        env:
          COSIGN_EXPERIMENTAL: "1"
        run: |
          while IFS= read -r img; do
            echo "Verifying: $img"
            cosign verify-attestation \
              --type slsaprovenance \
              --certificate-oidc-issuer https://token.actions.githubusercontent.com \
              --certificate-identity-regexp "^https://github.com/${{ github.repository }}/" \
              "$img"
          done < images.txt

The attestation-verify job only runs when images.txt exists in the repo root. That file should list OCI references produced by earlier build jobs (org/app@sha256:..., one per line). If the build pipeline isn't producing attestations yet, this job is skipped rather than failing the gate.

Post-merge provenance — the SLSA CLI that doesn't exist#

The most common error in SLSA recipes: a fictional standalone binary called slsa with a provenance create subcommand. No such binary exists in the SLSA GitHub Generator project. Any workflow that installs a binary called slsa and runs slsa provenance create will fail at the slsa invocation with "command not found."

What the SLSA GitHub Generator actually is#

SLSA GitHub Generator works through reusable workflows — you call them with uses: in your job definition, pinned to a version tag. The version pin is not optional: slsa-verifier validates that the provenance was generated by a trusted builder at a known version. A floating @main reference breaks the trust chain.

The correct structure is two jobs: build produces the image and its digest, then provenance calls the generator reusable workflow with those outputs.

L2 vs L3 — what's actually required#

The widespread claim that SLSA L3 requires hermetic builds (zero network access) is wrong. The SLSA spec is explicit: hermetic builds are considered in future directions, not a current L3 requirement.

Corrected post-merge workflow#

# .github/workflows/post-merge-attest.yml
name: post-merge-attest
on:
  push:
    branches: [main]
 
jobs:
  build:
    runs-on: ubuntu-22.04
    permissions:
      contents: read
      packages: write
    outputs:
      image: ${{ steps.push.outputs.image }}
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
 
      - name: Build and push
        id: push
        run: |
          IMAGE="ghcr.io/${{ github.repository }}/app"
          docker buildx build --platform linux/amd64 \
            -t "${IMAGE}:${{ github.sha }}" --push .
          DIGEST=$(docker inspect \
            --format='{{index .RepoDigests 0}}' \
            "${IMAGE}:${{ github.sha }}" | cut -d@ -f2)
          echo "image=${IMAGE}" >> "$GITHUB_OUTPUT"
          echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"
 
  provenance:
    needs: [build]
    permissions:
      actions: read
      id-token: write
      packages: write
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
    with:
      image: ${{ needs.build.outputs.image }}
      digest: ${{ needs.build.outputs.digest }}
    secrets:
      registry-username: ${{ github.actor }}
      registry-password: ${{ secrets.GITHUB_TOKEN }}
 
  sbom:
    needs: [build]
    runs-on: ubuntu-22.04
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
 
      - name: Install syft
        run: |
          curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh \
            | sh -s -- -b /usr/local/bin
 
      - name: Generate SPDX SBOM
        run: |
          IMAGE="${{ needs.build.outputs.image }}@${{ needs.build.outputs.digest }}"
          syft "$IMAGE" -o spdx-json > sbom.spdx.json
 
      - name: Publish as release asset
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh release create "attest-${{ github.sha }}" sbom.spdx.json \
            --title "Attestations ${{ github.sha }}" \
            --notes "SLSA L2 provenance + SPDX SBOM"

The provenance job delegates entirely to the SLSA GitHub Generator. It does not install any tooling. The reusable workflow handles the cosign attestation and pushes the provenance to the registry. The sbom job runs in parallel — it pulls the image by digest (immutable reference) and generates a dependency inventory with syft, then publishes it as a release asset alongside the provenance.

The SBOM as a release asset matters for a reason that isn't obvious: it creates a discoverable audit trail at a stable URL. An SBOM that exists only in a CI artifact expires. One attached to a release is queryable six months later when a CVE lands.

The tradeoff#

Policy gates add friction. Every Rego deny rule that fires creates a conversation with the PR author. Someone has to explain why execute_shell is on the banned list, why network: true requires a justification field. Maintaining the Rego files is real ongoing work — rules that are too strict get bypassed with conftest test || true; rules that are too permissive let the incidents through anyway.

The alternative is not frictionless — it is friction deferred. When the policy gate doesn't exist, every skill manifest that ships is an implicit approval of whatever the author happened to write. The tools key gets copy-pasted from examples without review. permissions.network: true gets left in from a development iteration nobody cleaned up. Nobody catches it because it was three lines inside a 400-line diff, and the CI was green.

The gate's friction is up-front and visible. The alternative's friction is retrospective and expensive. I have seen the retrospective version: a post-incident review where nobody could point to when the capability was authorized, only when it was first noticed in production.

OTel instrumentation — use the GenAI conventions#

The Python snippet circulating alongside these workflows uses custom metric names: llm.tokens, model.latency.ms, retrieval.hits. Those names work in isolation. They break when you connect to any backend that expects GenAI semantic convention attributes — Datadog, Grafana, and Uptrace all natively parse gen_ai.client.token.usage and gen_ai.usage.input_tokens; they do not natively parse your custom names.

The corrected snippet uses the stable vocabulary from opentelemetry-semantic-conventions:

from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.semconv._incubating.attributes import gen_ai_attributes
 
resource = Resource.create({"service.name": "agent-service", "service.version": "1.0.0"})
 
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer(__name__)
 
meter_provider = MeterProvider(
    resource=resource,
    metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter())],
)
metrics.set_meter_provider(meter_provider)
meter = metrics.get_meter("agent-service")
 
token_counter = meter.create_counter(
    name="gen_ai.client.token.usage",
    unit="{token}",
    description="Token usage by operation, model, and token type",
)
latency_hist = meter.create_histogram(
    name="gen_ai.client.operation.duration",
    unit="s",
    description="End-to-end model call duration",
)
error_counter = meter.create_counter(
    name="gen_ai.client.operation.errors",
    description="Unhandled errors by operation and model",
)
 
 
def traced_chat(model: str, messages: list[dict]) -> str:
    import time
 
    t0 = time.time()
    with tracer.start_as_current_span("chat") as span:
        span.set_attribute(gen_ai_attributes.GEN_AI_SYSTEM, "openai")
        span.set_attribute(gen_ai_attributes.GEN_AI_REQUEST_MODEL, model)
        try:
            response = client.chat.completions.create(model=model, messages=messages)
            elapsed = time.time() - t0
 
            input_tokens = response.usage.prompt_tokens
            output_tokens = response.usage.completion_tokens
 
            span.set_attribute(gen_ai_attributes.GEN_AI_USAGE_INPUT_TOKENS, input_tokens)
            span.set_attribute(gen_ai_attributes.GEN_AI_USAGE_OUTPUT_TOKENS, output_tokens)
 
            dims = {
                gen_ai_attributes.GEN_AI_SYSTEM: "openai",
                gen_ai_attributes.GEN_AI_REQUEST_MODEL: model,
            }
            for token_type, count in [("input", input_tokens), ("output", output_tokens)]:
                token_counter.add(count, {**dims, "gen_ai.token.type": token_type})
 
            latency_hist.record(elapsed, dims)
            return response.choices[0].message.content
        except Exception:
            error_counter.add(1, dims)
            raise

The metric names are now stable identifiers from the OpenTelemetry specification, not strings you own. When the specification evolves, the SDK package updates; your dashboard queries stay the same. For the full instrumentation pattern — OTEL Collector config, GCP export, and the span trace hierarchy — see LLM and Agent Observability with OpenTelemetry GenAI Conventions.

Rollout order#

Don't add all of this at once. The conftest gate on skill manifests has the highest immediate impact and the lowest implementation cost. SLSA provenance requires that the build pipeline is stable first — retrofitting it onto a build that changes frequently means the reusable workflow call breaks every time the image name or output format changes.

Start with conftest. The manifest-level capability gate is the gap most likely to cause an incident in a team that has never had one. It also fails loudly in the PR UI, which creates the policy conversation before the code ships rather than after.

SLSA provenance is the next priority. Add it when the build pipeline is stable. Pin the reusable workflow to a version tag — @v2.0.0, not @main. The attestation verification job in the pre-merge workflow is last because it depends on the provenance job in the post-merge workflow having run first.

If the agent's skill manifests aren't in your policy scope, the capability decisions are being made by whoever wrote the YAML last. The trust gate doesn't slow the team down; it makes the capability negotiation explicit instead of implicit.

Two reasons these setups fail most often: a fake slsa binary that does not exist, and a conftest rule aimed at a Markdown file it cannot parse. Both produce the appearance of a gate without the function of one. Fix the tooling first, then invest in the Rego rules.