Generative AI and language models

By the end, you can

• Explain, in plain language, that a language model generates text by predicting likely next tokens rather than retrieving a stored answer (AF-1).
• Explain why a fluent, confident answer is not by itself evidence that the answer is correct (AF-1).
• Give one example of a question a general-purpose language model cannot reliably answer without an external, current source (AF-1).

Before you start

This is Module 1, Lesson 2 of the AI Foundations course. It assumes Lesson 1's vocabulary — rule, data, pattern and model — and applies it to the specific kind of AI system most people now interact with daily: **generative AI**, and within it, the **large language model**, usually shortened to **LLM**. This lesson does not require any coding or technical setup.

Text broken into tokens, and one very specific job

A language model works with text broken into small pieces called **tokens** — roughly words or word-fragments. Google's own crash-course material puts the model's core job plainly: "a language model estimates the probability of a token or sequence of tokens occurring within a longer sequence of tokens." In practice, that means the model is trained on an enormous amount of existing text and learns which tokens tend to follow which other tokens.

When you ask an LLM a question, it is not looking your question up in a filing system. It is generating a reply one token at a time, each time choosing (or sampling from) the tokens its training makes most likely to come next, given everything written so far. That single mechanism — **pattern continuation over tokens** — is doing all of the work behind everything from a one-line reply to a multi-paragraph draft.

What pattern continuation buys you, and what it doesn't

This mechanism is genuinely useful for a wide range of everyday writing tasks, because so much everyday writing does follow learnable patterns: a polite reply, a summary of supplied notes, a first draft of a checklist, a rephrased version of a paragraph. That is squarely what generative AI is good at.

But the same mechanism has two limits that follow directly from how it works, not from a particular product being poorly built:

• **No automatic lookup.** A general-purpose language model has no built-in obligation to check a fact against a live source before writing it down. Unless it has been explicitly connected to a search tool, a database or some other external source, it is continuing a plausible-sounding pattern of text — which may or may not happen to match reality.
• **No built-in self-verification.** The model does not run an internal fact-checking pass on its own output before showing it to you. It produces the next likely token, and the next, and the next — fluency is a side effect of the mechanism, not a signal that a separate checking step has taken place.

Fluent is not the same as correct

These two limits combine into a specific, well-documented failure pattern. The Open Worldwide Application Security Project's generative-AI security effort (the OWASP Gen AI Security Project) describes it directly: "hallucinations occur when LLMs fill gaps in their training data using statistical patterns, without truly understanding the content" — producing "content that seems accurate but is fabricated." The US National Institute of Standards and Technology (NIST) has gone further and published a companion profile to its AI Risk Management Framework specifically covering risks unique to generative AI systems, precisely because failure modes like this do not show up the same way in older, non-generative software.

It helps to describe this in plain, non-anthropomorphic terms. The system did not "decide to lie," "get confused," or "believe" something false — those words describe a mind making a choice, and a language model is not doing that. A more accurate description: the system generated a statistically plausible continuation of the text so far, and that continuation happened not to match reality. Whether that continuation is checked against anything real is entirely a question of what happens next — a person reviewing it, or a separate tool verifying it — not something the generation step itself performs.

A worked example: a question with no reliable answer inside the model

Suppose you ask a general-purpose language model, with no connected tools, "what is today's exchange rate between the pound and the dollar?" The model was trained on text up to some point in the past and has no live connection to a currency feed. It can still produce a fluent, specific-sounding number — because producing a fluent, specific-sounding number is exactly what the pattern-continuation mechanism is built to do. That number is not evidence of anything; it is a plausible continuation of a sentence shape the model has seen many times before, current questions frequently followed by current-looking answers in its training text.

Contrast that with asking the same model to rephrase a paragraph you have supplied, or draft three subject-line options for an email you have written. Both of those tasks stay entirely inside "continue this pattern of text in a useful way" — there is no live external fact the model needs to have looked up, so the mechanism's limits do not bite in the same way.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. Introduction to Large Language Models — Google for Developers. Explains that a language model estimates the probability of a token or sequence of tokens occurring within a longer sequence, the basis of next-token text generation.
2. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile — NIST. A NIST companion profile to the AI Risk Management Framework, specifically addressing risks unique to generative AI systems.
3. LLM09:2025 Misinformation — OWASP Gen AI Security Project. Defines hallucination as an LLM generating content that seems accurate but is fabricated, filling gaps in training data using statistical patterns rather than verified fact.
4. Artificial Intelligence Risk Management Framework 1.0 — NIST AI Resource Center. Frames an AI system as an engineered system that generates outputs such as predictions — not an authoritative or self-checking decision-maker.