Sampling & Decoding Parameters

📌 The Probabilistic Nature of LLMs

Large Language Models do not look up answers in a database. Instead, they operate probabilistically: given a sequence of text, they calculate the probability of what the next word (or token) should be.

If an LLM always picked the absolute #1 most mathematically likely next word (a process called Greedy Decoding), its text would be incredibly repetitive, robotic, and boring. To fix this, developers adjust Sampling Parameters to inject controlled randomness into the model's choices.

📌 Temperature

Temperature is the most common parameter you will adjust. It controls the "creativity" or randomness of the model's predictions.

• Low Temperature (e.g., 0.1 - 0.3): Makes the model highly deterministic and focused. It almost always picks the most likely words. This is ideal for coding, data extraction, and factual Q&A where you want exact answers.
• High Temperature (e.g., 0.8 - 1.0): Flattens the probability curve, making the model more likely to pick less common words. This is ideal for creative writing, brainstorming, and writing poetry.

📌 Top-K and Top-P (Nucleus Sampling)

Before applying Temperature, the model often filters out the completely nonsensical words at the very bottom of its dictionary to prevent total gibberish. It does this using Top-K and Top-P.

• Top-K: Tells the model to only consider the top K most likely next words. For example, if Top-K = 40, the model completely ignores the 41st most likely word and below, even if the Temperature is high.
• Top-P (Nucleus Sampling): Tells the model to only consider the top words whose combined probabilities add up to P. For example, if Top-P = 0.90, the model takes the most likely words until their total probability reaches 90%, and discards the rest. This is generally preferred over Top-K because it dynamically adjusts depending on how "confident" the model is.