← Previous Next →

Choosing Context Windows

📌 The "Lost in the Middle" Problem

In previous lessons, we discussed the Context Window (an LLM's short-term memory). Modern models boast massive context windows—some up to 2 million tokens (about 1.5 million words). You might wonder: Why wouldn't you just use the biggest context window all the time?

Diagram showing a U-shaped recall graph over a long document, where the middle has low recall.
Diagram showing a U-shaped recall graph over a long document, where the middle has low recall.

Just because a model can accept 1 million tokens doesn't mean it remembers them perfectly. AI models suffer from a well-documented architectural issue called "Lost in the Middle".

  • Primacy Effect: They have near-perfect recall for facts placed at the very beginning of a prompt (like system instructions).
  • Recency Effect: They have near-perfect recall for facts placed at the very end of a prompt (like the final question you asked).
  • The Dip: They frequently "forget", ignore, or hallucinate facts buried in the middle of massive documents because the attention mechanism gets diluted across too many tokens.

How do we test this? The "Needle in a Haystack" Test: AI researchers test a model's true memory by taking a massive document (the haystack) and hiding a completely random, unrelated sentence in the middle (e.g., "The secret password to the server is ApplePie123"). They then ask the model what the password is. Older models completely fail to find the needle if it's placed in the middle 50% of the document.

📌 Matching Context to Use Cases

To optimize for cost, speed, and accuracy, you must choose a model whose context window matches your specific use case. Sending 100,000 tokens to a model when you only need 2,000 will result in incredibly slow response times and massive API bills.

Diagram showing an 8k bucket for chats, 128k bucket for PDFs, and 2M bucket for entire libraries.
Diagram showing an 8k bucket for chats, 128k bucket for PDFs, and 2M bucket for entire libraries.
  • Small Context (8k - 32k tokens): (Example: GPT-3.5, Llama 3 8B) Best for fast chatbots, customer service routing, writing single emails, or translating a webpage. These models are lightning-fast, extremely cheap, and have perfect recall because the prompt is short and focused.
  • Medium Context (128k tokens): (Example: GPT-4o, Claude 3.5 Sonnet) Best for analyzing a large PDF report, summarizing a 50-page financial document, or dropping in a specific folder of code for debugging. This is the "sweet spot" for most enterprise use cases today, balancing deep reasoning with reasonable costs.
  • Massive Context (1M - 2M tokens): (Example: Gemini 1.5 Pro) Best for cross-referencing an entire library of PDFs, analyzing hour-long videos natively, or ingesting entire codebases. These models are expensive and take significantly longer to generate a response (sometimes minutes), but they unlock entirely new workflows that were impossible a year ago.

📌 RAG vs. Massive Context

Because massive context windows are slow, expensive, and prone to the "Lost in the Middle" problem, engineers rarely stuff 1 million tokens into a prompt if they don't have to.

Instead, they use Retrieval-Augmented Generation (RAG). RAG acts like a search engine for your AI. Instead of giving the AI a 1-million-word book and asking it a question, RAG searches the book first, finds the 3 specific pages that contain the answer, and only sends those 3 pages (a few thousand tokens) to the LLM.

This hybrid approach (RAG + Small/Medium Context Models) is currently the industry standard for building fast, cheap, and highly accurate AI applications.

← Previous Next →