Context Window Management

Summary: Context window management is the set of techniques for fitting relevant information into an LLM's fixed context window during long-running tasks (agents, RAG, multi-turn dialog). Strategies include sliding windows, recursive summarization, hierarchical memory, and retrieval-augmented generation (RAG). Critical for autonomous agents running 100+ loop iterations.

The Core Problem

Context Window: Fixed size (4k–1M tokens depending on model)
Growing Input:  User query + history + tool results + docs + reasoning
                ↑
         Exceeds window → truncation → lost information
Model Context Window Year
GPT-3.5 4,096 2022
GPT-4 8,192 / 32,768 2023
Claude 3 200,000 2024
Gemini 1.5 1,000,000+ 2024
GPT-4o / o1 128,000 / 200,000 2024-25
Llama 3.1 128,000 2024

Key insight: Even with 1M tokens, naive accumulation fails — need active management.

Management Strategies

1. Sliding Window (FIFO)

def sliding_window(messages, max_tokens):
    while estimate_tokens(messages) > max_tokens:
        messages.pop(0)  # Drop oldest
    return messages
  • Pros: Simple, deterministic
  • Cons: Loses early instructions, system prompt, critical facts

2. Recursive Summarization

Diagram: (Mermaid diagram - view source for diagram code)

graph LR
    A[Turn 1-5] --> B[Summarize]
    C[Turn 6-10] --> D[Summarize]
    B --> E[Compressed History]
    D --> E
    E --> F[Current Context]
  • Pros: Preserves semantic content, bounded size
  • Cons: LLM call overhead, may lose nuance, error accumulation

3. Hierarchical Memory (Working + Archival)

Diagram: (Mermaid diagram - view source for diagram code)

graph TD
    A[Working Memory\n~4k tokens\nCurrent task] --> B[Archival Memory\nVector DB\nAll history]
    B --> C[Retrieval\nTop-k relevant]
    C --> A
    A --> D[Agent Loop]
  • Working: Active task context (instructions, recent turns, scratchpad)
  • Archival: Full history in vector DB, retrieved by relevance
  • Pros: Scales indefinitely, semantic retrieval
  • Cons: Retrieval latency, embedding quality dependency

4. RAG-Enhanced Context

Diagram: (Mermaid diagram - view source for diagram code)

sequenceDiagram
    participant User
    participant Agent
    participant VectorDB
    participant LLM
    User->>Agent: Query
    Agent->>VectorDB: Embed query → search
    VectorDB-->>Agent: Top-k chunks
    Agent->>LLM: Query + chunks + instructions
    LLM-->>Agent: Answer
  • Injects only relevant external knowledge
  • Reduces context pressure from docs

Agent-Specific Patterns

For Autonomous Agents (100+ loops)

Pattern Implementation Token Budget
System prompt Fixed, ~500 tokens Protected
Goal/Instructions Fixed, ~200 tokens Protected
Recent trajectory Last 5-10 turns, ~2k tokens Rolling
Tool results cache Last N results, compressed Rolling
Retrieved knowledge Top-3 from archival, ~1k tokens Per-turn

Codex Goals Pattern (OpenAI)

  • Persistent goal: Stored outside context, injected as system reminder
  • Session memory: Summarized per-session, loaded on resume
  • Result: Context stays focused on current subtask

Optimization Techniques

Token Counting (Accurate)

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = len(enc.encode(text))

Compression Strategies

Technique Compression Quality Loss
Remove whitespace/comments 10-20% None
Extract key facts 50-80% Low
LLM summarization 70-90% Medium
Embedding + retrieval 95%+ Depends on recall

FlashAttention Impact

  • Before: O(n²) memory → practical limit ~8k on 24GB GPU
  • After: Exact attention up to 32k-128k on same hardware
  • Enables: Longer context without approximation

Evaluation Metrics

Metric Target
Recall@k (archival retrieval) >0.9 for key facts
Task success rate >0.95 vs unlimited context baseline
Token efficiency <50% of max window used
Latency per turn <2s added overhead

Related Concepts

Sources