Hardware realities: what actually determines whether a model runs

By the end, you can

• Explain what has to fit in memory for a local model to run, and why a model's file size on disk is a practical stand-in for the RAM or VRAM it needs (LA-7).
• Distinguish CPU inference from GPU inference, and read a real runtime's own evidence of which one a running model is actually using (LA-7).
• Size a 7B-8B class model against a 16 GB-RAM machine's real headroom, using verified model-size figures rather than a single fixed rule of thumb (LA-7).

Before you start

This is Lesson 1 of 4 in Running models locally, the module that follows The model underneath and comes right before the course moves on to bounded task design. It assumes that module's vocabulary — tokens, context windows, the roles a runtime assembles into one request — but nothing about local setup yet; the setup lab is two lessons away, once this lesson and the next have given you the vocabulary for what you are actually sizing and choosing. No installation is required for this lesson. It does ask you to reason about real, current model-size figures, so a browser is useful.

RAM, VRAM and where a model's weights actually live

A model's weights — the learned numbers that make it behave the way it does — have to be loaded somewhere the processor doing the inference can reach quickly. There are two places that can be:

Which one a runtime actually uses is not a guess you have to make — Ollama's own `ollama ps` command shows it directly, in a `PROCESSOR` column reading `100% GPU` (weights entirely in VRAM), `100% CPU` (weights entirely in system RAM) or a split like `48%/52% CPU/GPU` for a model too large to fit in VRAM alone, offloaded partly to each. Lesson 4's lab has you read this column on a real running model; for now, the point to take is that "how much memory a model needs" is not one number — it depends on which pool of memory you're asking it to fit into, and a runtime will tell you, after the fact, which pool it actually used.

• RAM — the machine's main system memory, available to every program running on it, including a local model runtime.
• VRAM — a graphics card's own dedicated memory, separate from the machine's main RAM, reachable only by that GPU.

CPU inference vs GPU inference: the same weights, two different pools, two different speeds

Running a model — generating tokens from it, not training it — is called inference. The same set of weights can run through either a CPU or a GPU:

Neither path is "the" way to run a model locally — plenty of real local-agent work runs entirely on CPU, especially for smaller models or lower-throughput tasks, and the rest of this module gives you the vocabulary to judge, for a specific model and machine, whether that is fast enough for the job at hand. What both paths share is the same underlying inference engine: Ollama's own README documents its supported backend as "llama.cpp project founded by Georgi Gerganov" — Ollama is a friendlier, managed layer over the same llama.cpp inference engine this module also names directly, not two unrelated pieces of technology.

• CPU inference uses ordinary system RAM and works on essentially any machine, with no special drivers, but is markedly slower per token, since a general-purpose processor is not built for the kind of bulk matrix arithmetic a model's forward pass needs.
• GPU inference loads the weights into VRAM and uses the graphics card's own specialised hardware, which is dramatically faster per token — but only works if the weights (plus everything else a running request needs) actually fit in that card's VRAM, and only if the right driver stack is installed for it. Ollama's own Windows documentation is specific about this: it names a minimum NVIDIA driver version ("NVIDIA 551.61 or newer Drivers") and, for AMD cards, either a ROCm v7/HIP7-capable driver stack or a Vulkan-capable driver as a fallback. macOS instead names its GPU support by hardware family directly: "Apple M series (CPU and GPU support) or x86 (CPU only)" — meaning GPU acceleration on a Mac depends on which chip family it has, not on installing a separate driver.

A model's file size on disk is a working stand-in for the memory it needs

Here is the single most useful sizing heuristic in this lesson, and it comes with an honest caveat attached in its own source. llama.cpp's own documentation states it plainly: "As the models are currently fully loaded into memory, you will need adequate disk space to save them and sufficient RAM to load them. At the moment, memory and disk requirements are the same." The same document gives a concrete, worked table for Llama 3.1: the 8B model is 32.1 GB at its original precision but 4.9 GB once quantised to Q4_K_M; the 70B model drops from 280.9 GB to 43.1 GB at the same quant level; the 405B model drops from 1,625.1 GB to 249.1 GB.

That "at the moment" is doing real work — it is the document's own hedge that this equivalence reflects how these runtimes currently load a model (fully into memory, all at once), not a law of nature; a future runtime that streams weights from disk on demand could change the relationship. Treat the heuristic as reliable for how Ollama and llama.cpp work today, and re-check it if you read about a runtime doing something more exotic than a full in-memory load.

Two things this heuristic does not cover on its own, both of which this module returns to: quantisation is exactly why a model's original and quantised sizes differ so much (Lesson 2's job in full), and a running request needs memory beyond the weights themselves — the context window's contents (Lesson 1 of The model underneath) all have to sit in memory too, and Ollama's own context-length documentation is direct about the consequence: "Setting a larger context length will increase the amount of memory required to run a model."

A worked example: sizing a 7B-8B class model on a 16 GB-RAM machine

Take a genuinely common case: a builder with a 16 GB-RAM laptop, no dedicated GPU, wanting to run an 8B-class instruct model locally. Ollama's own library lists a live, checkable answer for one such model: the `llama3.1:8b` tag downloads at 4.9GB, quantised to Q4_K_M — the same 4.9 GB figure llama.cpp's own documentation independently gives for an 8B model at that quant level, which is a useful cross-check that the number is not a one-source fluke.

Applying the previous section's heuristic: loading that model's weights takes roughly 4.9 GB of RAM on a CPU-only machine. On a machine with no dedicated GPU, Ollama has no VRAM to measure, so its VRAM-tiered context-length default lands at its smallest tier — 4k tokens — unless raised. A 16 GB machine running this model at that default context has roughly 11 GB left after the weights alone, before accounting for the operating system, a browser, an editor or anything else already running, and before accounting for the extra memory a larger context window would add if the task needed the 64,000-token floor Ollama itself recommends for agent and tool-use workloads. That headroom is comfortable at the small default, tighter than it first looks once real context and everything else on the machine is counted, and worth actually watching (via `ollama ps`, covered in Lesson 4) rather than assuming.

Contrast the boundary case the same source table makes plain: a 70B model, even at the same Q4_K_M quantisation, is 43.1 GB — well beyond what a 16 GB machine can hold no matter how the memory is otherwise budgeted. No amount of closing background applications changes that; a builder targeting a machine like this one needs a materially smaller model, not a leaner desktop.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. All command examples and numeric tables from external sources are given as plain inline text, readable by a screen reader as ordinary text content. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. quantize tool README — Memory/Disk Requirements — ggml-org/llama.cpp. States that memory and disk requirements are currently the same for a fully-loaded model, with a worked Llama 3.1 table (8B, 70B, 405B original vs Q4_K_M sizes).
2. llama3.1:8b model page — Ollama. Lists the llama3.1:8b default tag as 4.9GB at Q4_K_M quantisation, cross-checked against llama.cpp's own figure for the same model and quant.
3. Context length — Ollama. States that setting a larger context length increases the memory required to run a model, and documents ollama ps output showing a running model's PROCESSOR split between CPU and GPU.
4. ollama/ollama README — Supported backends — Ollama (GitHub). Documents that Ollama's supported backend is llama.cpp, founded by Georgi Gerganov.