Hybrid AI Model Routing Is an Operating Boundary

The private model returned valid JSON, passed the schema, and still should not have been trusted.

It had copied a plausible identifier from the wrong row of a document. The syntax was clean. The value existed elsewhere on the page. A generic extraction benchmark would have called the response successful because every required field was present.

The business workflow cared about provenance, not presence.

That is the failure I design against when someone proposes adding an open model to reduce API calls. The model server is the easy box. The difficult part is deciding which requests may cross into it, what evidence must come back, when a hosted model gets another attempt, and which decisions remain outside model authority entirely.

This is part 1 of a four-part series:

1. Hybrid AI Model Routing Is an Operating Boundary
2. A Private GPT-OSS Service on Cloud Run
3. Cloud Run GPU or Model API: The Break-Even Math
4. Where Open Models Fit in DigDeepIQ's Acreage-Reporting Workflow

The series extends the private-inference lane in Local MCP and Private Open Model Infrastructure, the deployment discipline in Cloud Run GPU Sidecars Need Deployment Discipline, and the runtime analysis in The Faster Transformers Stack Behind GPT-OSS.

A Second Model Is Not a Fallback Strategy#

The first architecture people draw looks like this:

application -> cheap local model -> expensive API if local fails

The word fails hides the whole system.

Does failure mean an HTTP 500? A timeout? Invalid JSON? A tool call outside policy? A result that contradicts a deterministic calculation? A correct-looking value without page-and-row provenance? A response that passes syntax but falls below the evaluated accuracy threshold for this task class?

Without those definitions, the local model is not a controlled supplement. It is a new source of silent variance.

I use a model router as a policy service. It does not merely choose a URL. It owns task classification, model eligibility, request budgets, response validation, fallback, trace fields, and the final reason code.

The accepted output is still a candidate result. In a document workflow, deterministic rules or a person may own the final record. In an agent workflow, a separate permission layer may own whether a tool action can execute. Model quality does not grant authority.

Route Tasks, Not Prompts#

Prompt-level routing is too vague for production.

"Summarize this" can mean a disposable internal note or a customer-visible account decision. "Extract these fields" can mean finding a mailing address or populating a regulated record. The language is similar. The consequence is not.

I define task classes with explicit contracts:

tasks:
  document_page_classification:
    private_model_eligible: true
    max_input_tokens: 8000
    required_output_schema: page_classification_v2
    required_evidence:
      - page_number
      - supporting_text_span
    fallback_on:
      - timeout
      - invalid_schema
      - evidence_missing
      - confidence_below_0_92
 
  acreage_row_normalization:
    private_model_eligible: true
    model_output_is: candidate
    required_evidence:
      - source_page
      - source_row
      - source_cell_bounds
    deterministic_checks:
      - crop_code_is_known
      - acres_are_nonnegative
      - shares_are_in_range
    conflict_action: human_review
 
  sign_or_submit_report:
    private_model_eligible: false
    hosted_model_eligible: false
    authority: authenticated_human

This policy does more useful work than a table saying one model is "good for extraction." It names the versioned schema, evidence requirement, threshold, and conflict action. It also names a task no model may perform.

The Three-Lane Model#

Most hybrid systems need three lanes, not two.

The private model should not absorb work merely because the hosted API is expensive. It should absorb work for which the organization can define and measure acceptance.

This often makes seemingly boring tasks the best first targets:

• map a page into one of twelve known document types;
• convert an explanation into a fixed structured schema;
• draft an exception summary from already-verified facts;
• normalize a crop name against a supplied vocabulary without choosing the final code;
• rank likely supporting passages for a reviewer;
• label whether two rows deserve deterministic comparison.

The model is doing useful cognitive compression. It is not inventing the business truth.

An Application-Level Router#

The endpoint compatibility is convenient. GPT-OSS can be served through vLLM's OpenAI-compatible API, so an application can use the same client shape for private and hosted routes. Compatibility should reduce adapter code, not erase policy differences.

type Route = "private" | "hosted" | "manual";
 
type TaskRequest = {
  task: "page_classification" | "row_normalization" | "exception_analysis";
  inputTokens: number;
  containsRestrictedData: boolean;
  requiresVision: boolean;
  consequence: "low" | "reviewed" | "high";
};
 
type RouteDecision = {
  route: Route;
  reason:
    | "eligible_private_task"
    | "context_too_large"
    | "capability_required"
    | "restricted_by_policy"
    | "human_authority_required";
};
 
export function chooseRoute(request: TaskRequest): RouteDecision {
  if (request.consequence === "high") {
    return { route: "manual", reason: "human_authority_required" };
  }
 
  if (request.containsRestrictedData) {
    return { route: "private", reason: "restricted_by_policy" };
  }
 
  if (request.requiresVision || request.task === "exception_analysis") {
    return { route: "hosted", reason: "capability_required" };
  }
 
  if (request.inputTokens > 16_000) {
    return { route: "hosted", reason: "context_too_large" };
  }
 
  return { route: "private", reason: "eligible_private_task" };
}

The example is intentionally small. A production router also needs tenant policy, model revision, prompt revision, deadline, retry budget, and circuit-breaker state. The important property is that the choice produces a reason code. "The framework picked a model" is not enough evidence during an incident.

Fallback Must Preserve the First Failure#

A fallback can rescue the user while destroying the evidence.

If the private model emits invalid structured output and the hosted model succeeds, the final request looks healthy unless the trace preserves both attempts. Cost and quality analysis then lie. The organization sees only the successful hosted response, while the supposedly cheaper route may be failing twenty percent of the time.

I keep an attempt record like this:

{
  "task": "acreage_row_normalization",
  "policy_version": "hybrid-routing-v3",
  "attempts": [
    {
      "route": "private",
      "model": "openai/gpt-oss-20b",
      "model_revision": "sha256:example",
      "latency_ms": 1840,
      "input_tokens": 2140,
      "output_tokens": 188,
      "accepted": false,
      "reason": "source_cell_bounds_missing"
    },
    {
      "route": "hosted",
      "model": "hosted-model-snapshot",
      "latency_ms": 960,
      "input_tokens": 2280,
      "output_tokens": 203,
      "accepted": true,
      "reason": "schema_and_provenance_valid"
    }
  ],
  "final_disposition": "candidate_for_review"
}

That trace supports four different questions:

• Did the user get a usable result?
• Did the private route pass?
• What did the request actually cost?
• Is this task still eligible for private routing?

One success metric cannot answer all four.

Shadow Before Serving#

The first production milestone should not send private-model output to users.

Mirror eligible requests after the normal route has completed, remove or transform data according to policy, run the private model asynchronously, and compare results. This creates a workload-specific dataset without making the new model authoritative.

The comparison needs more than semantic similarity. For a structured document task, I would measure:

The denominator matters. If a private attempt costs one unit and fails, then a hosted fallback costs another unit, the request did not become cheaper because the first model had open weights.

Promotion Is Per Task and Per Version#

I do not promote "GPT-OSS" as a general capability. I promote a tuple:

task + model revision + runtime revision + prompt revision + validator revision

Changing any member can change behavior. A vLLM upgrade can alter parsing or scheduling. A prompt change can improve schema adherence and damage evidence selection. A validator change can make the same outputs appear more accurate. A quantization change can alter quality enough to matter on narrow identifiers.

A release record should therefore look like software:

release: acreage-normalization-private-v4
task: acreage_row_normalization
model: openai/gpt-oss-20b
model_digest: sha256:example
runtime: vllm-pinned-image@sha256:example
prompt: acreage-normalization-p7
validator: acreage-row-contract-v5
evaluation_set: acreage-golden-v12
promotion_thresholds:
  material_field_accuracy: ">= 0.995"
  unsupported_value_rate: "<= 0.001"
  schema_validity: ">= 0.999"
  fallback_rate: "<= 0.03"
rollback:
  action: disable_private_route
  owner: inference-on-call

Those numbers are illustrative, not DigDeepIQ production thresholds. The useful part is the shape: acceptance is explicit, versioned, and reversible.

The DigDeepIQ Business Boundary#

DigDeepIQ publicly describes a practical AI product built around acreage-reporting work. USDA's Risk Management Agency explains why this is a consequential workflow: acreage reports contain crop, acreage, planting, share, practice, type, and land-location information, and inaccurate reporting can affect coverage and premium.

That business context changes model routing.

A private model can help classify pages, normalize candidate values, draft reconciliation explanations, and reduce repeated hosted-model calls. It should not silently decide that a plausible field identifier belongs to a row, overwrite a conflicting value, or sign and submit a report. Those steps have different authority and evidence requirements.

The product advantage is not "we replaced an API." It is a better operating envelope:

• sensitive content can remain within a controlled project boundary for eligible tasks;
• repeated bounded work can use a predictable capacity pool;
• hosted models remain available for genuinely difficult cases;
• deterministic checks remain the source of truth where a specification exists;
• reviewers see provenance and conflicts instead of a model's confidence theater.

That is a stronger business story than model substitution because it connects cost control to correctness.

What I Would Build First#

Start with one task that already has reviewed examples and an exact output schema. Run it in shadow mode. Preserve both attempts. Price accepted results, not raw calls. Promote only after the model, prompt, runtime, and validator tuple passes the task-specific gate.

Then add a second task.

Do not begin with a universal router, a marketplace of models, or an agent that autonomously chooses its own authority. Begin where the organization can explain what correct means.

The private model becomes valuable when it disappears into a well-run workflow: bounded tasks go through it, evidence comes back, failures stay visible, difficult cases escalate, and no one confuses fluent output with final truth.

Next: A Private GPT-OSS Service on Cloud Run.

Sources#

• DigDeepIQ
• DigDeepIQ: About Us
• USDA RMA: Insurance Cycle
• USDA RMA: Acreage Crop Reporting Streamlining Initiative
• GPT-OSS 20B model card
• Cloud Run GPU support
• Cloud Run GPU inference best practices