Tag: inference

A production-oriented Cloud Run and vLLM design that keeps GPT-OSS private, separates routing from inference, and makes model loading, readiness, concurrency, and rollback explicit.

Reader outcome: Produced a deployable service contract for private OpenAI-compatible inference with IAM authentication, pinned artifacts, measured concurrency, and a scale-to-zero operating path.

Why Hugging Face's faster Transformers work matters beyond GPT-OSS, and how kernels, MXFP4, parallelism, KV cache, batching, and model loading change practical LLM runtime decisions.

Reader outcome: Mapped the GPT-OSS-era Transformers runtime features into concrete decisions about memory, compute, cache behavior, batching, and serving boundaries.

A practical runbook for scoring changed rows close to the data using Snowflake Streams and Tasks or BigQuery scheduled queries and remote models.

Reader outcome: Compared Snowflake and BigQuery patterns for scheduled in-warehouse inference, corrected CDC assumptions, and defined monitoring, grants, and deployment checks.