📌 The Problem of Repetition

Have you ever noticed an LLM getting stuck in a loop, repeating the same phrase (like "As an AI language model...") over and over? This happens because of how context windows work.

Once a model outputs a word, that word is added to its context window. Because the word is now in the prompt, the model becomes statistically more likely to output it again. If left unchecked, this can lead to disastrous loops. To solve this, we use two parameters: Frequency Penalty and Presence Penalty.

📌 Frequency Penalty (Encouraging Synonyms)

The Frequency Penalty penalizes a word based on how many times it has already appeared in the text.

Diagram showing a counter for a word tracking how many times it was used, with its probability dropping lower each time

If the model uses the word "furthermore", its probability drops slightly. If it uses it 5 more times, the probability drops massively. This penalty forces the model to dig into its vocabulary and find synonyms (like "additionally" or "moreover"), resulting in more natural, varied writing.

📌 Presence Penalty (Encouraging New Topics)

The Presence Penalty is a blunt instrument. It penalizes a word based on a simple True/False check: Has this word appeared at all?

Diagram showing a checkbox marking a topic as Present, instantly dropping its probability to encourage new ideas

Unlike Frequency Penalty, Presence Penalty does not care if a word was used 1 time or 100 times. The penalty is applied immediately upon the first use and stays flat. This is extremely useful for forcing the model to change the subject and brainstorm new ideas, rather than dwelling on the exact same topic for the entire response.

Frequency & Presence Penalties

📌 The Problem of Repetition

📌 Frequency Penalty (Encouraging Synonyms)

📌 Presence Penalty (Encouraging New Topics)

Quick Knowledge Check