Tag: data engineering

A Python architecture map for AI, data, and backend teams that need notebooks, prompts, evaluations, services, repositories, and infrastructure to stop collapsing into one folder.

Outcome: Defined a production Python layout for AI and data systems that separates experimentation, evaluation, domain logic, infrastructure adapters, and deployable service code.

A Python performance playbook for choosing data structures, profiling tools, vectorized libraries, JIT experiments, and concurrency changes from evidence instead of taste.

Outcome: Provided a performance triage workflow that asks for a measured bottleneck before changing algorithms, data structures, runtime flags, concurrency models, or native libraries.

A dbt on BigQuery starter kit for the parts that usually fail after the demo: raw loads without partition filters, snapshots with weak change detection, and CI that lets expensive SQL promote.

Outcome: Reader can scaffold a dbt and BigQuery project with manifest-backed incremental loads, timestamp-first snapshots, partitioned models, and a dry-run bytes gate before production promotion.

What an oversized iOS binary, a Linux linker error, and a SQL boundary teach about embedding DuckDB as the Parquet reader for languages without a mature native library.

Outcome: Reader can decide when DuckDB is the right Parquet path for a Swift or Zig project, configure the SPM and build.zig integrations correctly the first time, and avoid the binary-size and linker failures that the unconfigured path produces.

Why a precompiled-NIF fall-through on a less-common Linux target adds quiet minutes to a deploy, and what the borrowed-runtime pattern actually looks like for Elixir and Mojo.

Outcome: Reader can ship Parquet-reading Elixir without surprise source compilation in CI, recognize where Mojo's Python interop boundary is the bottleneck rather than Mojo itself, and know which DataFrame guarantees leak at the BEAM and PyArrow boundaries.

Why a default Go parquet.Read[T] call slurped a 1.4 GB file into 11 GB of resident memory, and the column-native Rust and Go patterns that replaced it.

Outcome: Reader can pick the streaming Parquet read path in Rust and Go, configure the compression-codec features explicitly, and avoid the eager-load anti-patterns that look fine in benchmarks and break in production.

A practical map from algorithmic complexity to software engineering, data pipelines, machine learning systems, and modern LLM architecture decisions.

Outcome: Connected classical algorithm analysis to production software, ML pipelines, RAG systems, model serving, and the trade-offs behind modern AI research.

A practical runbook for scoring changed rows close to the data using Snowflake Streams and Tasks or BigQuery scheduled queries and remote models.

Outcome: Compared Snowflake and BigQuery patterns for scheduled in-warehouse inference, corrected CDC assumptions, and defined monitoring, grants, and deployment checks.