← Previous Next →

Greedy Decoding vs Beam Search

📌 The Myopic Problem of Greedy Decoding

By default, if you set Temperature to 0, an LLM uses Greedy Decoding. This means that at every single step, the model looks at its options and picks the absolute most likely next word.

Flowchart showing a greedy agent taking the immediate highest probability path but ending up in a dead end
Flowchart showing a greedy agent taking the immediate highest probability path but ending up in a dead end

The problem with Greedy Decoding is that it ignores the big picture. Sometimes, picking a slightly sub-optimal word right now (e.g., a 40% probability word instead of a 60% probability word) unlocks a significantly better, more coherent phrasing down the line. Because Greedy Decoding never looks ahead, it misses these optimal paths entirely.

📌 The Solution: Beam Search

Beam Search is a decoding algorithm that solves this problem by exploring multiple paths simultaneously.

Flowchart showing multiple beams exploring a tree, finding a path that starts lower but ends with a much higher overall score
Flowchart showing multiple beams exploring a tree, finding a path that starts lower but ends with a much higher overall score

Instead of just keeping the single best word, Beam Search keeps track of a fixed number of "beams" (e.g., 3 separate paths). It branches out and predicts the next few words for each beam, calculates the cumulative probability of the entire phrase, and then picks the winner.

  • Pros: Results in significantly higher quality, more coherent text. It is especially useful in tasks like Machine Translation where grammatical structure requires looking ahead.
  • Cons: It is much more computationally expensive and slower than Greedy Decoding, because the model is effectively generating multiple sentences at once behind the scenes.
← Previous Next →