Tag: llm evaluation

This is one of the cleanest public pieces on agent design because it separates workflows from agents and keeps repeating the uncomfortable production lesson: start simple, add autonomy only when the task needs it, and make the tool interface inspectable.

Worth keeping next to any agent architecture work because it gives language for the tradeoff. Agents can improve task performance, but they buy that with latency, cost, and new failure modes.

Chip Huyen's agent writeup is useful as a systems map rather than a hype post. It treats agents as planning, tool use, memory, feedback, and environment design, which is the level where most production mistakes actually happen.

I would pair this with Anthropic's piece when deciding whether a workflow should stay deterministic or earn the complexity of agentic control.

This is a good companion link for grounding the agent conversation in the reasoning-to-acting lineage instead of treating tool use as a brand-new product category.

The useful angle for this site is not the vocabulary alone. It is the reminder that agent systems have to cross the boundary from thought traces to actions, and that boundary needs evaluation, tools, and constraints.

This is the DSPy link I would hand to someone who keeps saying "prompt engineering" when they really mean eval-driven optimization. The example is concrete, costed, and honest about train, validation, and holdout splits.

Especially useful because it makes LLM application work feel closer to data science: define a task, build examples, pick a metric, optimize, and check whether the gains survive outside the optimizer.

Part one is the practical on-ramp: a transcript-search use case, a RAG-ish starting point, and the moment DSPy becomes more than a nicer prompt template.

Good to keep because it explains why declarative signatures and modules matter before jumping into optimizer results. The concept lands better when the first problem is messy enough to deserve it.

This is a good reminder that eval improvements do not always come from a larger model. Sometimes the gain comes from rewriting the task boundary so the model can actually see the job.

The caveat is obvious but important: benchmark gains need translation into product behavior. Still, a 22 percent lift from prompt structure is worth pinning next to any agent eval work.