Context engineering: why 300 well-chosen tokens beat 100k noisy ones

An InfoQ architecture presentation makes the case that context quality is the new latency - and that most teams are over-indexing on window size instead of signal density.

In plain terms: Bigger context windows don't automatically mean better AI outputs. The engineers winning with LLMs in production are the ones carefully curating what goes into context - not just throwing everything in.

Baruch Sadogursky and Patrick Debois, in their InfoQ presentation (Aug. 14), make the case: retrieval, memory, and tool outputs that land in the context window are often redundant, contradictory, or simply irrelevant. Token budgets are finite in time and cost, not just in API limits - noisy context degrades reasoning quality and inflates latency. The alternative isn't a smaller model; it's a smarter harness.

Their architectural prescription: treat context as a resource to be managed, not a buffer to be filled. That means deliberate retrieval strategies (dense + sparse hybrid search, not just semantic similarity), structured formatting over raw text dumps, explicit compression of older context, and - critically - reasoning about what the model doesn't need to know for a given turn. They also introduce practical patterns: lazy-loaded skills, versioned context artifacts, and externalized memory banks as concrete implementations of this discipline.

Under the hood: The "300 tokens" framing is deliberately provocative - the point isn't a number, it's a discipline. In practice: avoid appending full documents when a cited passage suffices; summarize prior turns rather than replaying them verbatim; use structured slots (JSON schema) for tool results rather than free-text narration. Each reduces the signal-to-noise ratio the model has to reason through.

So what: As agentic systems grow - more tools, longer chains, more retrieved context - this problem compounds. Teams building on 200k+ context windows should audit how much of that window is doing actual work versus how much is noise they've normalized. The constraint is shifting from "can I fit it in?" to "should I put it in?"