Picking From The Spread

Two runs, two answers

You ask the same question twice and get noticeably different responses. Same wording, same tool, minutes apart.

The worrying interpretation is that something is unstable, or that one of the answers is the real one and the other is a glitch.

Neither. This is the system working exactly as designed, and once you see the mechanism it stops being unsettling.

Where it comes from

At each step the model produces a probability across every possible next chunk. Not one answer. A spread.

Something then has to pick one. That picking step is sampling, and it is deliberately not always taking the single most likely option.

One early divergence compounds. Choose a different opening word and everything after is conditioned on a different start, so two answers can end up substantially different from one small fork near the beginning.

Why not always take the most likely?

It sounds like always taking the top option would be better. It is not.

Always choosing the single most probable chunk produces text that is flat, repetitive, and prone to getting stuck in loops. Human writing is not the most predictable possible sequence of words, and text that is reads as lifeless.

So a controlled amount of randomness is introduced on purpose. The result is more natural, more varied, and less repetitive.

The dial

The amount of randomness is controlled by a setting usually called temperature.

Low. Strongly favours the most likely option. Repetitive, predictable, consistent between runs.

Middle. The default in most products. Natural-sounding, with real variation between runs.

High. Flattens the differences between options, so unlikely choices get picked more often. More surprising, and considerably more likely to go off the rails.

One clarification worth making: temperature does not make the model smarter or more creative in any meaningful sense. It changes how boldly it samples from the same distribution. High temperature does not produce better ideas, it produces less probable text, and those are not the same thing.

Where you can set it

In most consumer chat products you cannot, and it sits at a sensible middle value.

You get access when using an API, a developer playground, or a tool that exposes advanced settings.

But you can influence the effective variability without the setting, just by how you ask. A tightly specified request with a fixed output shape leaves much less room to diverge than an open one, because you have narrowed what a plausible answer looks like.