Sampling, determinism and structured output

By the end, you can

• Explain what temperature, top_p and top_k each do to a model's next-token choice, using a real local runtime's own parameter defaults (LA-6).
• Explain why a fixed seed and a temperature of 0 reduce run-to-run variation without making output fully reproducible, and why that matters for a bounded task (LA-6).
• Explain why models hallucinate as a structural consequence of how they are trained and evaluated, not a bug that better prompting alone removes (LA-6).
• Use a schema-constrained output mechanism to force a checkable response shape, and identify which parts of that response remain the model's unverified choice (LA-6).

Before you start

This is Lesson 3 of 3 in The model underneath, and the module's closing lesson — it carries this module's main practice exercise and knowledge check. It builds on Lesson 1's tokens and context-window budget and Lesson 2's context categories. It assumes Module 1's vocabulary. No running model is required, though the parameter values quoted here come from a real local runtime's own documentation.

From probabilities to one chosen token

At each step, a model does not simply "decide" the next token. It produces a probability distribution over every token in its vocabulary, and something has to turn that distribution into one chosen token. Three parameters, present in effectively every local runtime, control how that choice is made. Ollama's own Modelfile reference documents them, with defaults, this way:

These are not competing settings — runtimes commonly use filters like top-k and top-p to narrow the candidate pool and temperature to reshape the odds across what remains, though the exact order in which a runtime applies its sampling steps is an implementation detail that can differ between runtimes and change between versions.

• Temperature — "The temperature of the model. Increasing the temperature will make the model answer more creatively." Default: 0.8. Mechanically, temperature reshapes the probability distribution before a token is drawn — low values sharpen it toward the model's single most likely token, high values flatten it toward more even odds across many candidates.
• Top-k — "Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative." Default: 40. It restricts the draw to only the k most likely tokens before sampling.
• Top-p (nucleus sampling) — "Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text." Default: 0.9. Instead of a fixed count, it keeps the smallest set of top tokens whose combined probability reaches p.

A fixed seed narrows variation — it does not remove it

Ollama documents a fourth parameter for this same step: seed — "Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt," with a default of 0 (meaning unset, so each run draws its own randomness).

That description is the intent. In practice, even a fixed seed with matching parameters is only a strong reduction in variation, not an assurance of identical output every time. OpenAI's own cookbook guidance on its equivalent seed parameter is direct about the gap: matching the seed and parameters makes output "mostly deterministic," but full determinism is explicitly not assured, and the documentation points to backend and infrastructure changes as a source of drift even when a request looks identical to a previous one. The same caution applies locally — hardware differences, batching behavior and runtime version changes can all still shift results even at temperature 0 with a seed fixed.

For a bounded task, the practical implication is narrower than it sounds: a fixed seed and a low or zero temperature are worth setting whenever a task needs consistency, because they meaningfully cut down how much a rerun can vary — but do not build a task's stop conditions or its checks for a correct result around an assumption of byte-for-byte identical output. Build them around checking that the output actually meets what the task requires, every time, regardless of how consistent recent runs happened to look.

Why models hallucinate

None of the parameters above are the reason a model can state something false with complete confidence. That is a separate, structural property of how these models are built and measured, and it is worth naming plainly before this lesson gets to what can be done about it.

Research from OpenAI, examining this directly, argues that the incentive runs the wrong way during training and evaluation: "language models are optimized to be good test-takers, and guessing when uncertain improves test performance." A model that confidently guesses scores better, on average, than one that honestly says it does not know — the same distortion that rewards guessing over leaving an exam question blank. The same research reframes what a hallucination actually is: "Hallucinations need not be mysterious — they originate simply as errors in binary classification," the same kind of statistical error that shows up in any system trying to separate valid answers from invalid ones under uncertainty.

That framing matters for what comes next in this lesson: lowering temperature, fixing a seed, or constraining a response's shape all reduce variation — but none of them address why a model produces a fluent, confident wrong answer in the first place. A low-temperature hallucination is not a fixed hallucination. It is just a more consistent one.

Structured output constrains the shape, not the truth

llama.cpp documents a mechanism, GBNF, for forcing output into a defined structure: "GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp," letting a grammar require, in the documentation's own example, "valid JSON, or speak only in emojis." The constraint applies token by token during generation itself — a token that would break the grammar is never available to be chosen, so the output cannot help but conform to the shape the grammar defines.

Ollama documents a higher-level version of the same constrained-decoding idea: passing an actual JSON schema to the `format` parameter so that "Ollama now supports structured outputs making it possible to constrain a model's output to a specific format defined by a JSON schema." Rather than hoping the model remembers to close every bracket, the schema itself defines the only shapes the response is able to take.

Here is the distinction this lesson wants you to leave with: a schema forces the response to parse, and forces each field to have the right type and be present where required. It does not force any field's value to be true. A schema can require `"invoice_total"` to always hold a number — it cannot require that number to be the actual total on the invoice the model was given. Structured output solves a checkability problem, not a hallucination problem; the two are easy to conflate because a well-formed, schema-valid response looks trustworthy in a way a valid response is not automatically the same as an accurate one.

How systems compensate

No single control removes hallucination risk. A working combination, in roughly the order a bounded task should reach for them:

• Schema-constrained output — for anything downstream that needs to parse or act on the result, so a malformed response fails loudly rather than being silently misread.
• A fixed seed and a low or zero temperature — for tasks that need consistent behavior across reruns, understanding this narrows variation without making output fully reproducible.
• Grounding in tool or retrieved context (Lesson 2) rather than the model's own memory — asking the model to answer from a specific supplied document is a meaningfully different, checkable task from asking it to recall a fact unaided.
• Human review before anything the hallucination could actually damage — the same approval-boundary discipline Module 1 introduced. OpenAI's own guidance on this pause describes it as happening "so a person or policy can approve or reject a sensitive action" before that action takes effect — a check that catches a confidently wrong output no amount of sampling tuning would have caught on its own.

A worked example: a structured invoice extractor

A bounded task extracts fields from a short invoice-like text into JSON: `vendor_name` (string), `invoice_total` (number) and `due_date` (string). A JSON-schema-constrained run against a real invoice returns `{'vendor_name': 'Northwind Traders', 'invoice_total': 482.50, 'due_date': '2026-08-01'}`.

Annotated against this lesson: the response's shape — three fields present, correct types, valid JSON — is forced by the schema; no amount of model uncertainty can produce a response missing a field or holding a string where a number belongs. But each value's correctness is not forced by anything. `invoice_total` being a number is forced by the grammar; `482.50` being the actual total on this particular invoice is not — it is exactly as reliable as the model's reading of the source document, which is why a bounded task using this extractor still needs a check against the source text itself, not just a check that the JSON parses.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. Short field and JSON-value names appear as plain inline text (no special markup), read by a screen reader as ordinary text content. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. Modelfile reference — Ollama. Documents the temperature, top_k, top_p and seed PARAMETER options with their descriptions and default values (temperature 0.8, top_k 40, top_p 0.9, seed 0).
2. Reproducible outputs with the seed parameter — OpenAI. States that matching seed and parameters makes output "mostly deterministic," that full determinism is not assured, and that backend changes can still alter results even with a fixed seed.
3. GBNF grammars — ggml-org/llama.cpp. Documents GBNF as a formal-grammar format that constrains llama.cpp's output token by token to a defined structure such as valid JSON.
4. Structured outputs — Ollama. Documents passing a JSON schema to the format parameter so Ollama constrains a model's response to match that schema.
5. Why Language Models Hallucinate — Kalai, Nachum, Vempala and Zhang (OpenAI). Argues that hallucinations arise because training and evaluation reward confident guessing over acknowledging uncertainty, framing them as statistical classification errors rather than a mysterious flaw.
6. Guardrails and human review — OpenAI API documentation. Describes a human-review pause before a sensitive action takes effect — the compensating control this lesson applies to a confidently wrong, hallucinated output.