Embeddings and retrieval-augmented generation

By the end, you can

• Explain what an embedding is and how a runtime like Ollama's embedding API produces one, using its real, current request and response shapes (LA-9).
• Describe the RAG pipeline honestly — chunk, embed, store, retrieve, insert as retrieved context — and name two concrete retrieval failure modes (LA-9).
• Choose between plain keyword search and embeddings/vector retrieval for a stated memory task, and justify the choice (LA-9).

Before you start

This is Lesson 2 of 3 in State, memory and retrieval. It assumes Lesson 1's memory taxonomy — specifically that episodic and semantic memory need somewhere durable to live once they outgrow a single conversation's working memory — and The model underneath module's System, user, tool and retrieved context lesson, which already named retrieved context as one of the four kinds of content sharing a model's context window, without covering how something actually becomes "retrieved" in the first place. This lesson covers that mechanism directly: what an embedding is, how retrieval-augmented generation (RAG) uses one, honestly including where it fails, and when a much simpler tool does the same job better. No running model or install is required — one section offers a single optional hands-on command for learners who completed the Running models locally module's setup lab, and nothing in this module depends on running it — and every API shape shown here is real and current.

What an embedding actually is

An embedding is a fixed-length list of numbers — a vector — produced from a piece of text, positioned so that texts with similar meaning end up with similar (numerically close) vectors and unrelated texts end up far apart. That "similar meaning ends up numerically close" account is this lesson's own working simplification, stated as such rather than quoted from either cited source — the cited API documentation defines the mechanics (what goes in, what comes back), not the semantics — but it is the working assumption every retrieval system built on embeddings relies on, and the next section makes it concrete with real numbers rather than leaving it on faith.

Ollama's own API documents the mechanism plainly. A request to `/api/embed` takes exactly two required fields: "`model`: name of model to generate embeddings from" and "`input`: text or list of text to generate embeddings for." The response carries back an `embeddings` field — a list of vectors, one per input, shown in the documentation's own example (itself abbreviated — a real vector from this model family runs to hundreds of numbers) as `[[0.010071029, -0.0017594862, 0.05007221, 0.04692972, 0.054916814, 0.008599704, 0.105441414, -0.025878139, 0.12958129, 0.031952348]]` for a single input, with more than one vector returned when `input` was itself a list. The documentation is specific about one more detail worth knowing before relying on it: `truncate`, "truncates the end of each input to fit within context length. Returns error if false and context length is exceeded" — an embedding model has its own context limit, and a long piece of text handed to it can be silently cut down to fit unless you check for that.

One detail worth naming directly, because it trips builders moving from a chat model to embeddings for the first time: an embedding model is usually its own, separate, purpose-built model — not the same model generating your agent's replies. Ollama's own library lists `all-minilm`, described on its own page as an "embedding models on very large sentence level datasets" model that "can only be used to generate embeddings" — it cannot hold a conversation or call a tool at all, and its default download is a genuinely small 46MB, reflecting a narrower, more specialized job than a general chat model like the ones this course's earlier labs used.

Seeing "numerically close" in actual numbers

To make closeness concrete at a size you can eyeball, shrink the idea to toy scale. Suppose a query embeds to `[0.9, 0.1, 0.0]`, one stored note embeds to `[0.8, 0.2, 0.1]`, and another embeds to `[-0.1, 0.9, 0.4]`. You can see the pattern by looking: the first note's numbers point the same general way as the query's — large first component, small second — while the second note's do not. **Cosine similarity**, the comparison named in the retrieval step below, is the standard way to score exactly that visual impression: it measures how closely two vectors point in the same direction, on a scale where 1.0 means perfectly aligned and values near 0 mean unrelated. Here the first note scores roughly 0.98 against the query and the second roughly 0.0 — the number agreeing with what your eye already saw.

Real embeddings do exactly this with hundreds of numbers instead of three — far too many to eyeball, which is why the comparison is computed rather than read, and why the toy version above is the honest place to build the intuition. If you completed the Running models locally module's setup lab, you can produce real embeddings yourself in one command — this is optional hands-on, not one of this course's labs, and nothing later in this module depends on running it. Pull the embedding model first (`ollama pull all-minilm` — the small 46MB download described above), check your server answers with `ollama ps` (the same thirty-second check Tool use and the agent loop's lab opens with), then run:

```shell curl http://localhost:11434/api/embed -d '{ "model": "all-minilm", "input": ["a cat sat on the mat", "a kitten rested on the rug", "quarterly financial projections"] }' ```

Run while writing this lesson against a real local Ollama server, this exact command returned three vectors of 384 numbers each. Computing cosine similarity across them scored the two related sentences at about 0.62 against each other, and the unrelated third sentence at about −0.04 against the first — the same shape as the toy example, at real scale, from real sentences no one hand-picked numbers for. One honest note on reading the output: the raw 384 numbers themselves are not individually meaningful, and eyeballing them side by side tells you nothing — at real scale the comparison has to be computed (a few lines of Python, if you want to try it yourself; Tool use and the agent loop's lab has a check-you-have-Python section if you've not run Python before). The toy example is where the eyeball check works; the real output is why retrieval systems compute it instead.

RAG mechanics, honestly

Retrieval-augmented generation is the pattern of using embeddings to pull relevant material into a model's context before it answers, rather than expecting the model to already know it. The pipeline, stated plainly rather than glossed over:

1. **Chunk** — split a document collection into smaller pieces, since embedding an entire long document as one vector loses the specificity a query needs. 2. **Embed** — run each chunk through an embedding model (Ollama's `/api/embed`, or an equivalent), producing one vector per chunk. 3. **Store** — keep each chunk's text alongside its vector, in a vector database, a vector-capable extension of an existing database, or even just an in-memory list for a small enough collection. 4. **Retrieve** — at query time, embed the query itself with the same embedding model, then find the stored chunks whose vectors are closest to the query's vector (commonly by cosine similarity), returning the top few — "top-k." 5. **Insert as retrieved context** — hand those top-k chunks to the model as part of its context window. This is precisely The model underneath module's retrieved context category from the outside; RAG is simply the mechanism that decided what to retrieve.

What this pipeline does not do, and where it honestly fails, matters as much as the five steps above. Anthropic's own research on this exact problem states the core issue plainly: "Traditional RAG solutions remove context when encoding information, which often results in the system failing to retrieve the relevant information from the knowledge base" — a chunk embedded in isolation can lose enough of its surrounding meaning that a genuinely relevant query still doesn't surface it. This is not a rare edge case dressed up as a caveat: the same research measured a top-20-chunk retrieval failure rate of 5.7% for standard RAG before any mitigation. That figure's scope is worth stating precisely rather than reading it as a universal constant: it is the baseline for the *top-performing* embedding configuration Anthropic tested (Gemini Text 004), from an evaluation run across the specific document domains the research covered — codebases, fiction, arXiv papers and science papers. Roughly one in eighteen retrievals missed a chunk that should have been found — with a top-tier embedding model, on that evaluation's own data. A small local embedding model like the 46MB `all-minilm` shown earlier should be expected to miss more, not less. The honest takeaway is the direction, not the decimal: retrieval misses are a measured, normal outcome of a correctly built pipeline, and a memory design that assumes every retrieval is complete is trusting exactly what this evidence says to verify.

A second, related failure worth naming even though it is this lesson's own reasoned judgment rather than a quote from either cited source: a **stale chunk**. An embedding captures what a chunk of text said at the moment it was embedded. If the underlying source changes afterward — a policy is updated, a fact becomes wrong — and the store is never re-embedded, the stored vector and the text next to it keep confidently returning outdated information, with nothing in the retrieval mechanism itself flagging that anything has gone stale. Retrieval accuracy and retrieval freshness are two separate problems, and a system that solves the first says nothing about the second.

When plain search beats a vector database

Retrieval doesn't have to mean embeddings at all. SQLite ships a full-text search module, FTS5, described in its own documentation as "an SQLite virtual table module that provides full-text search functionality to database applications" — supporting the `MATCH` operator, ranking results "by relevance" using a built-in ranking function, and prefix, phrase and proximity (`NEAR`) queries, all with no embedding model, no vector index and no extra infrastructure beyond SQLite itself. For a scope smaller than a database at all, a plain keyword tool such as `grep` or `ripgrep` over a folder of notes does the same essential job with even less setup.

Neither this lesson nor its sources declare a universal winner between plain search and vector retrieval — the honest answer is that it depends on what the memory actually holds and what a query against it looks like:

For this course's own scoped, single-agent memory — the kind Lesson 3 designs hygiene rules for — that tradeoff usually favors plain search first: a personal agent's own semantic and episodic memory is exactly the small, mostly-exact-match, low-infrastructure case FTS5 or even `grep` was built for, and reaching for a vector database by default, before the collection or the query pattern actually demands it, adds a dependency and a re-embedding maintenance burden a scoped memory store frequently does not need to pay.

• **Plain search tends to win** when matches are lexical — an exact order ID, an error code, a phrase the user is likely to type close to verbatim — when the collection is small enough that a scoped local agent's own memory store realistically stays (hundreds to low thousands of entries, not a company-wide knowledge base), and when the value of zero extra infrastructure (no embedding model to run, no vector index to keep in sync with a changing store) outweighs anything vector search would add.
• **Vector retrieval tends to earn its cost** when a query and its matching content don't share the same words — a paraphrase, a related concept expressed differently — and when the collection has grown large enough that scanning it directly, even with an indexed keyword search, stops being the bottleneck-free option it is at a smaller scale.

A worked example: choosing for a small personal research assistant

A local research assistant keeps roughly 200 notes the user has explicitly asked it to remember — short facts, task reminders, a handful of longer summaries. Two realistic queries: "what did I decide about the Q3 budget" (the user is likely to phrase this close to how the note itself is worded) and "find anything relevant to convincing a skeptical stakeholder" (a concept that might be phrased many different ways across the notes, if it appears at all).

For a 200-note collection, SQLite FTS5 handles the first query well and cheaply — no embedding model to run, and the query's own wording is likely close enough to the note's wording for a keyword match to find it directly. The second query is exactly the paraphrase-shaped case vector retrieval is built for — "skeptical stakeholder" might never appear verbatim in a note about "the client kept pushing back in the review," which is nonetheless the relevant one. A reasonable design for this scale keeps FTS5 as the default and adds embeddings only where the collection or the query pattern has actually shown plain search falling short — not as an assumed starting point.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. All code, JSON and prose examples appear as plain inline or block text, readable and copyable by assistive technology and keyboard-only users. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. ollama/ollama docs/api.md — Ollama (GitHub). Documents the /api/embed endpoint's model and input fields, its embeddings response field, batching multiple inputs, and the truncate parameter's context-length behavior.
2. all-minilm — Ollama. Describes all-minilm as an embedding model that "can only be used to generate embeddings," distinct from a chat/instruct model, with a small (46MB) default download size.
3. Introducing Contextual Retrieval — Anthropic. States that traditional RAG removes context when encoding information, which often causes retrieval to miss relevant information, and reports a 5.7% baseline top-20-chunk retrieval failure rate for its top-performing tested embedding configuration (Gemini Text 004) across the four document domains its evaluation covered, before its own contextual-embeddings mitigation.
4. FTS5 — SQLite. Describes FTS5 as an SQLite virtual table module providing full-text search, supporting the MATCH operator, ranking by relevance, and prefix, phrase and NEAR queries.