Context Engineering: Why Larger Windows Haven't Made Agents More Reliable

LLM context windows have exploded from an early 8K to 128K, 1M, and even 2M tokens. Many developers once held an optimistic assumption: if the window is large enough to hold an entire project’s codebase, couldn’t we simply dump every file and log into the agent to deliver complex architectural refactoring with ease?

Real-world experience quickly shattered this illusion. Even when models claim to support massive context windows, after dozens of tool calls and debugging turns, agents still routinely forget initial architectural constraints, run in circles on failed approaches, or even overwrite freshly written code based on stale memory.

Simply expanding raw physical capacity does not guarantee stability in long-horizon tasks. In modern coding agent workflows, the deciding factor between success and failure has shifted from tweaking prompts to context engineering—the practice of dynamically maintaining high-quality working memory.

The Context Bloat Trap: Attention Limits and Degradation

Computer scientist Andrej Karpathy once proposed a classic architectural metaphor: think of the LLM as the CPU in a new computing architecture, and the context window as its RAM.

Just as an operating system must rigorously manage virtual memory to prevent crashes, an agent’s attention is a scarce, expensive computational resource. Massive context windows lull developers into letting their guard down—allowing raw logs and irrelevant code to saturate the context, triggering two fatal forms of cognitive degradation:

1. Long-Range Retrieval Degradation (“Lost in the Middle”): Research demonstrates that long-context models excel at retrieving information at the beginning and end of the context, but retrieval performance traces a sharp U, bottoming out for content caught in the middle. As conversation turns accumulate and tool outputs pile up, system constraints and interface details sandwiched in the middle are prone to being completely overlooked.
2. Accuracy Degradation: Chroma Research’s Context Rot study shows that a model’s accuracy consistently degrades as input length grows, and in non-uniform ways—even when task difficulty stays fixed and only the input gets longer. Irrelevant tokens dilute attention weights, causing the model to miss critical boundary conditions or hallucinate.

Three Common Symptoms of Context Rot

The progressive degradation of system performance over extended tasks is known as context rot. In daily development, its most common manifestations include:

• Goal Drift: As debugging logs accumulate, the model may lose track of initial constraints, making unauthorized edits to untouched core files or violating the project’s established code style.
• Error Self-Reinforcement: If stack traces from failed prior attempts aren’t cleaned up in time, subsequent reasoning loops will treat flawed code structures as ground truth, trapping the agent in an endless cycle of ineffective fixes.
• Stale State Persistence: Even after file contents have been updated on disk, earlier read logs of the old version linger in the context window. The model then applies edits based on outdated line numbers and variable names, blindly overwriting code the agent itself has just written.

Information beneficial to the current task—such as project conventions, the latest code, and verified results—can be viewed as “signal.” Irrelevant logs, stale file contents, and redundant messages are “noise.” The Signal-to-Noise Ratio (SNR) describes the balance between the two: the higher the effective information and the lower the interference, the easier it is for the model to stay focused on what matters.

A core objective of context engineering is maximizing this ratio, expressed conceptually as:

$\text{SNR}_{\text{context}} = \frac{\sum \text{Relevant Task Information}}{\sum \text{Irrelevant Noise, Stale State, and Redundant Logs}}$

In practice, that means giving the model only the minimal and necessary information at the right time.

Practice 1: Constrain Read Scope and Tool Outputs

In coding agent workflows, raw outputs from tool calls—such as hundreds of lines of build logs from npm test, verbose git log dumps, or broad codebase searches—are often the largest token sinkholes. Letting these unfiltered dumps accumulate often exhausts the attention budget within a dozen or so turns.

Developers can start by cutting unnecessary inputs:

• Just-in-Time Retrieval: Avoid pasting entire modules or massive files into the prompt upfront. Instruct the agent to search for relevant symbols first, read only the necessary line ranges, and expand its reading scope only as concrete clues emerge.
• Constrain Tool Outputs: When tests or builds generate extensive logs, save the full output to disk and let the agent inspect only the failed assertions, exit codes, and relevant error messages first, referencing the complete log only when necessary. Relying solely on the tail of the output risks missing root causes; collapsing logs in the UI does not reduce the tokens sent to the model.
• Extract Key Summaries: When an agent identifies the three interface signatures it needs in a 2,000-line file, ask it to summarize those signatures and their file paths for subsequent steps or handoffs. However, appending a summary does not automatically purge prior read logs; reclaiming context space still requires leveraging compaction/cleaning mechanisms provided by tools or starting a fresh session.

Practice 2: Tiered Memory and File Persistence

When solving complex architectural problems, human engineers don’t keep every nuance in active memory—they take notes. Relying solely on a fragile, rot-prone chat history as the single source of truth is the root cause of failure in long-horizon tasks.

The most robust approach is implementing tiered memory and persisting task state to local files:

Write-Verification Separation: Keeping Flawed Memories from Solidifying

Without proper verification, erroneous assumptions readily accumulate in an agent’s memory. High-reliability agents should enforce the Write-Verification Separation principle:

Provisional hypotheses generated during an agent’s reasoning must never be treated as long-term facts. Only after code passes actual compilation, unit tests, or explicit verification steps should conclusions be committed to persistent memory, preventing false assumptions from becoming entrenched.

When to Compact Context vs. When to Start a Fresh Session

Turn count makes a poor threshold for deciding when to reset: a single turn might contain only a few sentences or hundreds of lines of tool output. A better signal is whether the agent begins retrying ruled-out approaches, referencing outdated code versions, or forgetting established constraints. Transitions between task phases are also natural moments to prune and refocus context.

Context compaction is best suited when a task remains continuous but the session has accumulated excessive detail. For instance, Claude Code’s /compact condenses conversation history into a summary to shrink active context and keep the session moving. This is not equivalent to starting a fresh session, nor should you assume it will produce structured Markdown handoff files for you; after compaction, you should still verify that crucial constraints and action items were preserved.

Starting a fresh session is ideal when stale assumptions persistently pollute reasoning or when work shifts to a distinct objective. Before resetting, write the current goal, verified progress, ruled-out approaches, and next steps into task.md, and commit or stage your code changes. Letting the agent read the handoff file alongside the latest codebase eliminates noise from past conversations; because handoff summaries themselves can contain omissions or drift, they must always be cross-checked against actual files and test results.

Advanced Practices: Information Firewalls and Prompt Caching Discipline

Understanding the architectural mechanisms of modern LLMs can further enhance reliability and cost efficiency in daily development:

Sub-agents as Information Firewalls

When dealing with expansive exploratory tasks—such as chasing elusive bugs or conducting repo-wide searches—a single session is easily contaminated by voluminous exploration logs.

Deploying independent sub-agents is the most effective solution:

• Establish an Information Firewall: A research sub-agent consumes tens of thousands of tokens inspecting files within its isolated window, returning only a concise summary of a few hundred words to the parent agent. The massive exploratory noise is discarded when the sub-agent terminates, allowing the main workflow to remain in a clean, focused, high-level planning state.
• Independent Review (Reviewer Pattern): Have a reviewer agent inspect changes against task specifications, project conventions, Git diffs, and test results without loading the coding agent’s internal thought processes or debugging history. This mitigates confirmation bias, though reviewers can still make mistakes—judgments must ultimately be grounded in code and verification results.

Prefix Invariance Discipline for Prompt Caching

In prompt engineering, API caching (Prompt Caching / Prefix Caching) drastically reduces latency and slashes API costs by 60% to 80%. However, prefix caching relies on exact character-by-character prefix matching: any modification to the prefix instantly invalidates all cached tokens downstream.

In day-to-day use and rule configuration, adhere to the following principles:

1. Tiered Structural Ordering: Anchor the most static system configurations and project rules (e.g., AGENTS.md) at the very top of the prompt, append conversation history sequentially below, and place volatile context (such as the current timestamp or one-off instructions) at the very end.
2. Append-Only Progression: Maintain an append-only discipline throughout the conversation, avoiding dynamic timestamp insertions at the start of historical messages or tweaking the formatting of earlier turns.
3. Batch Resets Over Incremental Tweaks: When context needs cleaning, perform a deliberate, one-time session compaction to rebuild the cache rather than making frequent, minor edits across turns, which continuously break cache continuity.

Conclusion: From Prompt Crafting to Working Memory Management

As AI-assisted development evolves toward long-horizon autonomous tasks, the engineer’s role is undergoing a profound transformation: we are no longer merely “prompt writers,” but architects of an agent’s execution environment and working memory.

Rather than holding out hope for hypothetical future models with infinite, distortion-free windows, we should practice rigorous context discipline in our daily workflows: retrieve just-in-time, compact or reset sessions when appropriate, persist state to disk, and leverage sub-agents as information firewalls. Mastering context engineering is what unlocks consistent, dependable agent performance across complex, long-horizon tasks.