Limits, Context Windows, & Costs

📌 The Context Window (Short-Term Memory)

An LLM's Context Window is its short-term memory. It defines exactly how many tokens the model can "hold in its head" at any given time during a single API call.

Think of it like a conveyor belt. If a model has a 128,000 token context window, it can read about a 300-page book in one go. But if you paste a 400-page book into the prompt, the model will "forget" the first 100 pages. As new tokens enter from the right (your newest messages), the oldest tokens fall off the cliff on the left.

📌 Input vs Output Costs

API providers (like OpenAI, Anthropic, and Google) charge you based on how many tokens you use. However, not all tokens are priced equally.

• Input Tokens (Cheap): The text you send to the model (your prompt). Providers charge very little for this (e.g., $2.50 per 1 million tokens) because modern GPUs can process your entire prompt in parallel, all at once.
• Output Tokens (Expensive): The text the model generates back to you. This is usually 3x to 5x more expensive (e.g., $10.00 per 1 million tokens). Why? Because LLMs are autoregressive. They must generate text sequentially, one single token at a time, calculating the probability of the next word over and over again.

📌 Managing Costs in Production

Because every token costs money, developers use several strategies to keep bills low:

• Context Windowing: Instead of sending the entire 100-message chat history every time the user types "hello", developers slice off older messages, keeping only the last 10 relevant messages.
• Prompt Caching: Many providers now offer caching. If you send the same massive system prompt or document thousands of times a day, the provider caches those input tokens, reducing the price by up to 50-80%.
• Model Routing: Using a massive, expensive model (like GPT-4) to summarize a simple paragraph is overkill. Developers route simple tasks to cheap, fast models (like GPT-4o-mini), and save the expensive models for complex reasoning.