A Private GPT-OSS Service on Cloud Run

The first request reached the container while the model was still loading.

The port was open. The health check was green. The GPU had not finished receiving the weights. The application retried, Cloud Run created another instance, and the team paid for two cold starts while the user waited on both.

That is not a model-quality failure. It is an inference-service contract failure.

This is part 2 of a four-part series:

1. Hybrid AI Model Routing Is an Operating Boundary
2. A Private GPT-OSS Service on Cloud Run
3. Cloud Run GPU or Model API: The Break-Even Math
4. Where Open Models Fit in DigDeepIQ's Acreage-Reporting Workflow

The target here is not a private chat UI. Cloud Run GPU Sidecars Need Deployment Discipline already covers the Ollama plus Open WebUI shape. This service is application infrastructure: a private, OpenAI-compatible model endpoint called by another service.

Why GPT-OSS 20B and vLLM#

GPT-OSS 20B is a practical baseline for this experiment because its model card gives the deployment useful properties:

• Apache 2.0 licensing;
• tool use and structured-output capabilities;
• configurable reasoning effort;
• an MXFP4 path designed to fit in approximately 16 GB of GPU memory;
• documented Transformers, Ollama, and vLLM usage;
• an OpenAI-compatible HTTP surface when served with vLLM.

"Approximately 16 GB" is not a promise that every production configuration fits comfortably on a 16 GB device. Weights are not the whole memory budget. The runtime also needs activations, kernels, and KV cache. Context length and concurrent sequences can consume the headroom faster than the parameter count suggests.

Cloud Run's NVIDIA L4 gives the service 24 GB of VRAM. That makes it a plausible first target for GPT-OSS 20B, but only a load test can decide the supported context and concurrency for the actual workload.

I prefer vLLM here because the service needs online inference rather than a desktop model manager. vLLM owns continuous batching, paged KV-cache behavior, an OpenAI-compatible API, structured-output support, and serving controls that can be benchmarked directly. Ollama remains excellent for local interaction and the Open WebUI pattern. The runtime should match the workload.

Separate the Gateway From the GPU#

I would deploy two Cloud Run services.

The gateway is a normal CPU service. It authenticates the caller, applies tenant and task policy, strips disallowed content, chooses a route, validates the result, records the trace, and calls a hosted fallback when permitted.

The inference service owns one job: load one pinned model revision and answer compatible generation requests.

Putting the policy gateway into the GPU container couples cheap routing work to expensive capacity. It also makes every policy deployment restart the model. Separating them gives the model service a slower, deliberate release cadence and lets the gateway scale independently.

The Inference Service Contract#

Before the deployment command, I write the contract:

service:
  name: private-gpt-oss-20b
  ingress: internal-and-cloud-load-balancing
  authentication: cloud-run-iam
  caller_service_account: model-router
 
artifact:
  model: openai/gpt-oss-20b
  model_revision: pinned-hugging-face-commit
  runtime_image: pinned-vllm-image-digest
  response_format: harmony-through-supported-runtime
 
capacity:
  gpu: nvidia-l4
  gpu_count: 1
  cpu: 4
  memory: 32Gi
  min_instances: 0
  max_instances: 2
  initial_concurrency: 4
  max_model_length: 16384
 
readiness:
  port_open_is_not_ready: true
  model_loaded_probe_required: true
  warmup_request_required: true
 
release:
  no_floating_tags: true
  traffic_migration: canary
  rollback: previous_cloud_run_revision

The values are starting points, not universal settings. I use 32 GiB of instance memory for runtime and model-loading headroom even though the L4 service minimum is 16 GiB. I start concurrency at four because copying 80 from a general HTTP example would be fiction. The load test changes it.

Pin the Runtime and Model#

Two mutable artifacts can change the result: the runtime image and the model repository.

A production Dockerfile should use a validated digest:

ARG VLLM_IMAGE
FROM ${VLLM_IMAGE}
 
ENV HF_HOME=/models/huggingface \
    VLLM_LOGGING_LEVEL=INFO
 
# The runtime image and model revision are supplied by the release manifest.
# Do not replace them with floating `latest` or an unpinned model branch.

The build invocation supplies something shaped like:

docker build \
  --build-arg "VLLM_IMAGE=vllm/vllm-openai@sha256:VALIDATED_DIGEST" \
  --tag "$REGION-docker.pkg.dev/$PROJECT/inference/gpt-oss-runtime:$RELEASE" \
  --file Dockerfile.inference .

The digest in the example is deliberately a placeholder. An article should not bless a runtime revision I have not tested against the reader's driver, GPU, model revision, and parser behavior.

Pin the Hugging Face model revision as well. A model identifier names a repository; a commit identifies the artifact evaluated for release.

Model Storage Is a Cold-Start Decision#

Google's Cloud Run GPU guidance recommends container images for smaller model artifacts and Cloud Storage-based loading for larger models. It specifically calls out images as best suited to models below roughly 10 GB. GPT-OSS 20B is large enough that I would treat model loading as a designed subsystem rather than casually baking it into an ever-growing application image.

The practical choices are:

For Cloud Storage model loading, Google's current best-practices guidance calls for Direct VPC egress with Private Google Access when optimizing the path. The model service account should get read access to one model bucket or prefix, not broad project storage authority.

I would create a release manifest next to the artifact:

{
  "model": "openai/gpt-oss-20b",
  "source_revision": "PINNED_COMMIT",
  "artifact_uri": "gs://private-models/gpt-oss-20b/PINNED_COMMIT/",
  "artifact_sha256_manifest": "sha256sums.txt",
  "runtime_image": "REGION-docker.pkg.dev/PROJECT/inference/vllm@sha256:VALIDATED",
  "evaluation_release": "hybrid-routing-golden-v4"
}

The service refuses to become ready if the checksums or expected revision do not match.

A Cloud Run Deployment Shape#

Once the image and model artifact are prepared, the deployment is conceptually small:

gcloud run deploy private-gpt-oss-20b \
  --project "$PROJECT" \
  --region "$REGION" \
  --image "$PINNED_RUNTIME_IMAGE" \
  --execution-environment gen2 \
  --no-allow-unauthenticated \
  --service-account "$MODEL_SERVICE_ACCOUNT" \
  --cpu 4 \
  --memory 32Gi \
  --gpu 1 \
  --gpu-type nvidia-l4 \
  --no-gpu-zonal-redundancy \
  --no-cpu-throttling \
  --min 0 \
  --max 2 \
  --concurrency 4 \
  --timeout 600 \
  --command vllm \
  --args "serve,$MODEL_PATH,--served-model-name,gpt-oss-20b,--host,0.0.0.0,--port,8080,--max-model-len,16384,--gpu-memory-utilization,0.90"

This is an architecture example, not a blind copy-paste deployment. Current vLLM arguments, the model-loading path, reasoning parser, tool parser, and quantization behavior must be verified against the pinned runtime. The final command belongs in the evaluated release manifest.

The service is private. Cloud Run IAM checks the caller. The gateway service account gets roles/run.invoker on this service and nothing broader merely because both workloads live in the same project.

Call It With a Service Identity#

The gateway obtains an identity token for the Cloud Run audience and sends a normal OpenAI-compatible request:

from typing import Any
 
import google.auth.transport.requests
import google.oauth2.id_token
import requests
 
 
def chat_completion(
    service_url: str,
    messages: list[dict[str, str]],
) -> dict[str, Any]:
    auth_request = google.auth.transport.requests.Request()
    token = google.oauth2.id_token.fetch_id_token(auth_request, service_url)
 
    response = requests.post(
        f"{service_url}/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
        },
        json={
            "model": "gpt-oss-20b",
            "messages": messages,
            "temperature": 0,
            "max_tokens": 500,
        },
        timeout=90,
    )
    response.raise_for_status()
    return response.json()

For streaming, token refresh, retries, and cancellation, I would wrap this in the gateway's normal HTTP client policy. I would not put hosted-provider credentials into the GPU service. The model container has no reason to know the fallback route.

Readiness Means the Model Can Answer#

A TCP startup probe can be useful, but it is insufficient when the runtime opens the port before weights are ready. Google's guidance explicitly warns that Ollama can do this; the broader lesson applies to any serving stack.

I want two levels:

1. A startup probe that does not route traffic until model initialization completes.
2. A release warmup that sends one representative structured request and validates the response.

The warmup should be cheap and deterministic:

{
  "model": "gpt-oss-20b",
  "messages": [
    {
      "role": "user",
      "content": "Return JSON with status set to ready and integer schema_version set to 1."
    }
  ],
  "temperature": 0,
  "max_tokens": 40
}

The release does not receive traffic because a process exists. It receives traffic after the endpoint demonstrates the response contract.

Concurrency Is the Economic Lever#

Cloud Run does not autoscale GPU services from GPU utilization. It uses request concurrency and CPU signals. If service concurrency is too high, requests wait inside an overloaded instance and p95 latency collapses. If it is too low, Cloud Run creates more GPU instances than the model needs.

Google recommends load testing different concurrency settings. I would test a matrix:

For each cell, capture time to first token, total latency, aggregate output tokens per second, GPU memory, queue time, schema validity, and error rate.

The result may show that concurrency four is wrong. Good. The starting number existed to make the test possible, not to win an argument.

Scale to Zero Versus Minimum Instances#

min=0 is an economic choice with a latency cost.

Cloud Run can start the GPU infrastructure quickly, but the service still has to obtain and load model weights. A user-facing synchronous route may not tolerate that cold path. A background document pipeline may tolerate it easily.

I would use one of three modes:

• Scale to zero: development, shadow traffic, scheduled batches, and low-frequency tasks.
• Scheduled warm capacity: set a minimum instance before a known processing window, then return to zero.
• Always warm: only after traffic and latency economics justify the monthly floor.

Scheduled warm capacity is underrated. Agricultural and insurance workflows can have seasonal and deadline-driven peaks. A platform does not need to pay the same idle bill in a quiet week as it pays during a known reporting window.

The Release Checklist#

Before the first private route receives user traffic:

• model and runtime are pinned by digest or commit;
• the model artifact checksum is verified at startup;
• the service is not unauthenticated;
• only the gateway identity can invoke it;
• readiness proves the model can answer, not only open a port;
• the maximum context is explicit;
• concurrency has a representative load-test result;
• maximum instances fit regional GPU quota;
• traces include model, runtime, prompt, validator, latency, and token counts;
• the hosted fallback has a separate deadline and budget;
• rollback means moving traffic to the previous revision or disabling the route;
• the cost ledger counts failed private attempts and hosted fallbacks.

The GPU service is ready when its behavior can be reproduced, observed, and rolled back. A successful gcloud run deploy proves only that Cloud Run accepted a revision.

Next: Cloud Run GPU or Model API: The Break-Even Math.

Sources#

• GPT-OSS 20B model card
• OpenAI GPT-OSS repository
• vLLM documentation
• Cloud Run GPU support
• Cloud Run GPU inference best practices
• Cloud Run: Run AI inference on GPUs
• Cloud Run authentication overview