Tag: inference

4 entries tagged "inference" — 3 posts, 1 link.

Posts

Jul 16, 2026 — 9 min — Platform & AI

A Private GPT-OSS Service on Cloud Run

A production-oriented Cloud Run and vLLM design that keeps GPT-OSS private, separates routing from inference, and makes model loading, readiness, concurrency, and rollback explicit.

cloud run gpt-oss

Mar 12, 2026 — 9 min — Platform & AI

The Faster Transformers Stack Behind GPT-OSS

Why Hugging Face's faster Transformers work matters beyond GPT-OSS, and how kernels, MXFP4, parallelism, KV cache, batching, and model loading change practical LLM runtime decisions.

transformers gpt-oss

Oct 27, 2025 — 10 min — Platform & AI

In-Warehouse Inference on Snowflake and BigQuery

A practical runbook for scoring changed rows close to the data using Snowflake Streams and Tasks or BigQuery scheduled queries and remote models.

snowflake bigquery

Links

Articlemedium.comMar 19, 2026Permalink

Scaling Inference To Billions of Users And Agents

Federico Iezzi, Google Cloud

This is useful because it connects agent adoption to inference architecture. Agents do not make one call; they fan out across planning, retrieval, tool use, retries, and evaluation, which changes the serving math quickly.

Worth keeping as a scale reference for Google Cloud AI work. The product question is where inference cost becomes a feature constraint rather than a backend detail.

All tags