Temperature, Top-P, and LLM Parameters Explained

How Language Models Generate Text

Before understanding the parameters, you need to understand the generation process. A language model predicts the next token (word or sub-word) by computing a probability distribution over its entire vocabulary. For a model with a 100,000-token vocabulary, each generation step produces 100,000 probabilities, one for each possible next token. The model then selects a token from this distribution, adds it to the sequence, and repeats the process for the next token.

The sampling parameters control how the model selects from this probability distribution. Without any sampling modifications (temperature = 1.0, no top-p, no penalties), the model samples randomly from the full distribution weighted by the probabilities. The most likely token is selected most often, but less likely tokens are selected occasionally, which produces natural variation in the output. The sampling parameters modify this selection process to make it more or less random, more or less diverse, and more or less repetitive.

Temperature

Temperature is the most important sampling parameter. It scales the probability distribution before sampling. Mathematically, the probabilities are divided by the temperature value before the softmax function is applied, which reshapes the distribution.

Temperature 0.0 (or very near 0) makes the model deterministic. It always selects the highest-probability token at each step. This means the same prompt produces the same output every time (or very nearly so, as implementation details can introduce minor variation). Use temperature 0 for tasks where consistency is essential: data extraction, classification, code generation, and any task where you want the same input to produce the same output across runs.

Temperature 0.1 to 0.3 introduces slight variation while keeping the output focused and coherent. This range works well for analytical tasks, summarization, and question answering where you want the model to be mostly deterministic but occasionally choose a slightly different phrasing or approach. Most production applications use this range.

Temperature 0.4 to 0.7 is the "general purpose" range. There is enough randomness for natural-sounding text without the output becoming unfocused or erratic. This works for conversational applications, content generation, and tasks where some variety is desirable but accuracy is still important.

Temperature 0.8 to 1.0 produces notably varied, creative output. Each run produces meaningfully different text, which is useful for brainstorming, creative writing, and generating diverse options. The trade-off is reduced consistency: the model may produce excellent output on one run and mediocre output on the next, because higher temperature means lower-probability (and potentially lower-quality) tokens get selected more often.

Temperature above 1.0 flattens the probability distribution so much that very unlikely tokens get selected frequently. The output becomes increasingly random, incoherent, and error-prone. Values above 1.2 are rarely useful for any practical purpose. Some models cap temperature at 2.0; others allow higher values but produce unintelligible output above 1.5.

Top-P (Nucleus Sampling)

Top-p is an alternative approach to controlling randomness that operates differently from temperature. Instead of scaling all probabilities, top-p limits the model to considering only the tokens whose cumulative probability adds up to the specified threshold. A top-p of 0.9 means: sort all tokens by probability, add them up from highest to lowest, and stop when you reach 90% cumulative probability. The model samples only from these tokens, ignoring the remaining 10% of the distribution (the long tail of very unlikely tokens).

Top-p 0.1 restricts the model to only the most probable tokens, producing very focused, deterministic output (similar to low temperature). Top-p 0.9 includes most of the probability mass, allowing natural variation while still excluding the extremely unlikely tokens that would produce errors. Top-p 1.0 includes all tokens, effectively disabling the filter.

The practical difference between temperature and top-p is subtle. Temperature uniformly scales all probabilities, making even very unlikely tokens more or less likely. Top-p makes a binary decision: tokens are either in the candidate set or excluded entirely. This means top-p is better at preventing the very unlikely tokens (typos, wrong-language tokens, nonsense) that high temperature can produce, while still allowing variation among the reasonable tokens.

Most practitioners use either temperature or top-p, not both simultaneously. Using both adds complexity without proportional benefit, and the interaction between them can produce unexpected results. OpenAI's documentation recommends adjusting one and keeping the other at its default value.

Frequency and Presence Penalties

Frequency penalty reduces the probability of tokens proportional to how many times they have already appeared in the output. A token that has appeared 3 times gets a larger penalty than one that appeared once. This parameter fights repetitive output, where the model loops back to the same phrases, sentences, or ideas. A frequency penalty of 0.2 to 0.5 is enough to reduce repetition without causing the model to awkwardly avoid common words. Higher values (above 1.0) cause the model to actively avoid any repeated words, which produces unnatural prose that strains to find synonyms for common terms.

Presence penalty applies a fixed penalty to any token that has appeared at all in the output, regardless of how many times. It encourages the model to introduce new topics and concepts rather than revisiting ones it has already discussed. A presence penalty of 0.2 to 0.5 gently encourages topic diversity. Higher values push the model to cover new ground with each paragraph, which is useful for brainstorming but counterproductive for tasks that require depth on a single topic.

For most production applications, leave both penalties at 0 or set them to a low value (0.1 to 0.3). The default behavior of modern frontier models produces sufficiently non-repetitive output for most tasks. Only increase these penalties when you observe specific repetition problems in your output.

Max Tokens

Max tokens limits the length of the model's response. This is not a sampling parameter (it does not affect which tokens are selected, only how many), but it directly affects output quality and cost.

Setting max tokens too low truncates useful output. The model generates as much as it can within the limit, then stops mid-sentence or mid-section. This produces incomplete, low-quality responses that confuse users and break downstream parsers. Always set max tokens high enough to accommodate the longest response your application might need, with a buffer for variation.

Setting max tokens too high wastes money if you are paying per token (some models charge for the max tokens allocation rather than actual tokens generated). It can also encourage the model to pad its response with unnecessary content to fill the available space, producing verbose output when a concise answer would be better.

For structured output (JSON), you can estimate the expected length more precisely and set a tighter limit. For free-form text, a useful heuristic is: estimate the expected response length, multiply by 1.5 for a safety margin, and use that as your max tokens. Monitor actual token usage in production and adjust.

Recommended Settings by Use Case

Data extraction and classification: Temperature 0, top-p 1.0, no penalties, max tokens matched to expected output size. You want the same input to produce the same output every time, with no variation.

Code generation: Temperature 0 to 0.2, top-p 1.0, no penalties. Code needs to be correct, not creative. Low temperature keeps the model focused on the most likely (and usually most correct) code patterns.

Analytical writing (summaries, reports, explanations): Temperature 0.2 to 0.4, top-p 0.95, frequency penalty 0.1 to 0.2. Enough variation for natural-sounding prose, enough consistency for reliable quality.

Conversational AI (chatbots, assistants): Temperature 0.5 to 0.7, top-p 0.9, frequency penalty 0.2. Responses should sound natural and varied across conversations, not robotic and repetitive.

Creative writing and brainstorming: Temperature 0.7 to 1.0, top-p 0.95, presence penalty 0.3 to 0.5. High variation, topic diversity, and creative exploration. Accept that some outputs will be lower quality in exchange for more interesting and surprising results.

Multi-turn reasoning and chain-of-thought: Temperature 0 to 0.1. Reasoning should be deterministic. Creative variation in a reasoning chain introduces errors rather than interesting alternatives.