The same prompt is not a repeatable answer

One answer is one sample

Take an invented example. A fictional advice service asks a model to sort 200 incoming messages into three categories. It records the labels and moves on. A week later the same prompt and the same model name are run over the same 200 messages. Nine labels change. Nobody edited the prompt.

Calling that a bug is the wrong diagnosis. The model is a sampling system. A response is one draw from a distribution conditioned on the input, the model, and the machinery that served the request. A second draw is not a replay.

That matters before any decision where a single output becomes a record: a classification, an extraction, a summary that someone signs, or a test that expects an exact string.

Temperature zero is not a switch for determinism

Most APIs expose temperature and top-p to shape sampling. Lower temperature concentrates the choice on more probable tokens. At temperature zero, the usual description is greedy decoding: always take the highest-probability token. That sounds deterministic, and mathematically it is described that way.

Thinking Machines Lab's September 2025 post makes the practical point directly: with temperature at zero, "LLM APIs are still not deterministic in practice." The post covers hosted APIs and also local inference libraries such as vLLM and SGLang. Its explanation is not that sampling is random at zero, but that the arithmetic and scheduling around it are not fixed.

So the first correction is a habit: do not read temperature zero as a guarantee. It narrows the output distribution. It does not freeze the result.

Your request shares a batch

The same post explains batch invariance. A forward pass is deterministic given exactly the same inputs, but the server's inputs include the other requests in the same batch. Changing the batch size can change the arithmetic used for one row, so a request's output can depend on what else was running at the time.

That is why a busy API can return a different answer without the provider announcing a model change. The request you sent was identical. The batch around it was not.

vLLM's reproducibility page says the same thing in operational terms: it "does not guarantee the reproducibility of the results by default, for the sake of performance." It offers deterministic scheduling offline and batch invariance online, and still limits reproducibility to the same hardware and the same vLLM version. Interpretation: reproducibility is a configured property of a particular stack, not a property of a prompt.

Provider settings are part of the input

A model name is not a frozen artefact. Behind one name, the provider may change backend configuration, routing, or the model version it serves. OpenAI's API specification describes the seed parameter as a best-effort request for deterministic sampling and then says plainly that determinism is not guaranteed. It also documents a response field intended to show when backend changes might affect determinism. A seed is therefore useful evidence of intent, not a receipt that the next run will match.

Google's Gemini API documentation is more direct about one changing control. It states that Gemini 4 and Gemini 3.6+ models ignore custom temperature, top_p and top_k values and return deprecation warning headers. A prompt carried over from an older model may look identical while the sampling controls it relied on no longer apply.

Judgment: any record that stores only the prompt text is incomplete. The model identifier, the provider, the parameters, the system instructions, the tool configuration, and the date are part of the input too.

Record the run, not just the answer

A repeatable workflow does not need every reply to be identical. It needs the variation to be visible and bounded. A practical record has five parts:

This is the same discipline as retrieval is not verification, applied to the run instead of the source.

• **Identity.** Store the full model identifier, the provider, the API version or date, and any version label the provider exposes.
• **Inputs.** Store the system prompt, user prompt, parameters, tool definitions, and retrieved context alongside the result. A prompt that references a retrieval step is not reproducible without the retrieved text.
• **Raw output.** Keep the unparsed response. A parsed JSON object loses the wording and structure that might explain a difference.
• **Repeat evidence.** Run the same input several times and record the spread. For an evaluation, report the number of runs and an interval, not one score. Evan Miller's paper treats evaluations as experiments and gives formulas for that uncertainty; the point is that a single score overstates what one run can tell you.
• **Separate checks.** Put deterministic assertions around the edges: the response parses, required fields exist, values fall inside allowed sets, and links resolve. Then treat the model's judgement as a measured quantity rather than a fixed fact.

What repeatability cannot prove

A deterministic answer can still be wrong. If a model reliably misreads a contract clause, exact repetition makes the error easier to reproduce, not less wrong. Reproducibility answers "why did this run differ from that one?" It does not answer "is this correct?"

There is also a cost to over-constraining. Drafting and brainstorming benefit from variation; forcing one output makes the tool less useful and can hide better options. The question is whether the output becomes a decision or a record. For a record, measure and record the variation. For a draft, accept that it varies and review the content.

What we would do next

Pick one workflow where a model output becomes a decision. Add the model identifier, parameters and raw response to its record if they are missing. Run the same input ten times and count how often the decision changes. If it changes, decide whether the variation is acceptable for that decision. If it does not change on ten runs, write down that this is evidence from ten runs, not a guarantee. Then pin the model version where the provider allows it, and re-run the check when the model name or provider settings change.

This article was drafted with AI assistance and checked against the cited pages on 22 October 2026. It reports no measured result from a production system, no private incident, and no testimonial.

Sources and limits

This article synthesises the sources below into a practical explanation. It is not a security standard, legal advice, or a guarantee that guidance current at review time still applies — check the review date above against your own situation.

1. Defeating Nondeterminism in LLM Inference — Thinking Machines Lab. Explains sampling, states that temperature zero makes greedy decoding theoretically deterministic yet LLM APIs are still not deterministic in practice, and shows that inference is not batch-invariant, so one request's output can depend on the batch it shares with other requests.
2. OpenAI API specification (openapi.yaml) — OpenAI. Documents the seed request parameter as a best-effort attempt at deterministic sampling, states that determinism is not guaranteed, and documents a response field intended to show when backend configuration changes might affect determinism.
3. Gemini API: Text generation — Google. States that Gemini 4 and Gemini 3.6+ models ignore custom temperature, top_p and top_k values and return deprecation warning headers.
4. Reproducibility — vLLM. States that vLLM does not guarantee reproducible results by default, documents batch invariance and deterministic scheduling options, and limits reproducibility to the same hardware and the same vLLM version.
5. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Evan Miller. Treats evaluations as experiments and gives formulas for analysing evaluation data, measuring differences between models, and planning an evaluation, including uncertainty.

More from Server Attic

• 2026-10-12

  A model card measures someone else's task

  A model card and a public benchmark report someone else's test. Freeze a small set of your own cases, name the failure you refuse, and re-run that set before a model does the job unattended.

• 2026-08-03

  Don't ask AI if it's right

  A confident answer is not evidence. Build a small verification plan around the claim, the consequence of being wrong, and the source or test that can actually settle it.

• 2026-10-02

  Retrieval is not verification

  Adding a search step to an AI assistant makes answers look grounded. Retrieval decides what text the model sees; it does not check that the text is right, current, or actually supports the claim.