Key Model Parameters

Context windows, temperature, top-p sampling, and system prompts — understanding these parameters gives you meaningful control over LLM behaviour in production.

Context windows

The context window is the amount of text (measured in tokens) the model can "see" at once when generating a response — including the system prompt, conversation history, and any documents you pass in. Context windows range from 4k tokens for older models to over 1 million tokens for some frontier models. Longer context enables more sophisticated tasks but increases latency and cost.

Temperature

Temperature controls the randomness of generation. At temperature 0, the model always picks the most probable next token (deterministic, repetitive). At higher temperatures (0.7-1.0), lower-probability tokens are sampled more often, producing more varied and creative outputs. For factual tasks, use low temperature. For creative tasks, experiment with higher values.

Top-p (nucleus) sampling

Rather than sampling from all possible tokens, top-p restricts generation to the smallest set of tokens whose cumulative probability exceeds p. At top-p 0.9, the model samples from the top 90% of probability mass. This prevents extremely unlikely tokens while maintaining diversity. Temperature and top-p interact — typically you adjust one or the other, not both simultaneously.

System prompts

System prompts are instructions provided to the model before the conversation begins, typically by the developer rather than the end user. They set the model persona, establish rules and constraints, provide context about the deployment environment, and shape how the model should respond. Well-designed system prompts are a significant lever for production quality.

Temperature is the most frequently misunderstood parameter. Setting it to 0 does not make the model more accurate — it makes it more repetitive. Accuracy is a function of the model and prompt, not temperature.