Choosing your first local model

Start with the job, not the model

It's tempting to open a model library, sort by download count, and grab whatever's popular. That works out worse than starting from the other end: name the job first — a narrow, bounded task like summarising short documents, drafting replies to a fixed set of message types, or classifying text into a handful of categories — and let the job's difficulty and your hardware decide the model, not the other way round. A small model asked to do a small job well will usually serve you better than a large model you can barely run and don't actually need.

The number that decides whether it runs at all: memory

Before quality or speed, one number gates everything else: does the model fit in memory. Local model files are commonly distributed in a quantised format — a version of the model stored at lower numeric precision than it was trained at, which shrinks the file. The tool used to produce these files documents a direct, practical consequence of how local inference works today: "As the models are currently fully loaded into memory, you will need adequate disk space to save them and sufficient RAM to load them. At the moment, memory and disk requirements are the same." In plain terms: a model's download size is not just a disk-space number — it is a genuinely reliable estimate of the RAM (or VRAM, on a GPU) you need to run it at all.

A quick sizing example, before you download anything

Take a well-known example: an 8-billion-parameter model at a commonly-used quantisation level. Ollama's own model listing shows the llama3.1:8b model's default download as 4.9GB at `Q4_K_M` quantisation — a figure that also matches the independent size table published by the quantisation tool itself for the same model and quant level. If your machine has, say, 16GB of RAM, a model in the 4-5GB range leaves meaningful headroom for your operating system, your browser and whatever else is running; a model in the 12-14GB range does not. That single comparison — model file size versus your machine's actual free memory, not its total memory — is the first filter, before you've weighed anything about quality.

Instruct, not base, for almost everyone starting out

Most model families ship two broad kinds of file: a base model, trained to continue text, and an instruct (or "chat") model, further trained to follow direct instructions and hold a conversation. If your job looks like "ask it to do something and get a direct answer back" — which covers nearly every practical local-agent task — an instruct-tagged model is almost always the right starting choice. A base model is a reasonable pick only if you specifically intend to fine-tune it or build your own scaffolding around it, which is a deliberate, more advanced choice, not a default one.

A short checklist before you download

Four questions, in order, cover most of the decision for a first local model:

Answering these four honestly, in this order, rules out most of the bad first choices — an oversized model that swaps to disk and crawls, a base model that ignores your instructions, or a download that turns out to be twice your free memory — before you've spent any real time on it.

• What is the job, stated narrowly enough that "done" is checkable — not "help me with emails," but "draft a reply to a support message in a fixed tone"?
• How much RAM (or VRAM) can this model actually use, leaving headroom for everything else running on the machine — not the machine's total memory, its free memory?
• Does the model's listed download size, at a commonly-used quantisation level such as `Q4_K_M`, fit inside that free-memory budget, using the size-equals-memory-need relationship above?
• Is an instruct-tagged version of the model available for the job — the answer is almost always yes, for any well-known model family?

Where this article stops and the course starts

This article is deliberately a narrow decision aid, not a technical reference: fit the model in your actual free memory, pick the instruct variant for a direct-answer task, and treat the download size as your real memory estimate. It does not cover how to measure your own hardware's real headroom, how to choose between a smaller model at higher precision and a larger model at lower precision for the same memory budget, how to interpret a runtime's own evidence of CPU-versus-GPU inference, or how to capture real latency and memory numbers once a model is actually running — all of which the choice genuinely depends on once you move past a first pick.

Learning Harbour's Local AI Agents course covers exactly that depth, hands-on, in its "Running models locally" module: hardware realities and what has to fit in memory, a full worked quantisation ladder for one real model, a live install-and-run lab against a real local runtime, and a lab where you capture your own measured evidence rather than relying on someone else's numbers. If the sizing example above was useful, that module is where the same decision gets a properly worked, measured answer for your own machine — this article is the five-minute version that gets you to a reasonable first download, not a substitute for it.

Sources and limits

This article synthesises the sources below into a practical explanation. It is not a security standard, legal advice, or a guarantee that guidance current at review time still applies — check the review date above against your own situation.

1. quantize tool README — Memory/Disk Requirements — ggml-org/llama.cpp. States that memory and disk requirements are currently the same for a fully-loaded model, with a worked Llama 3.1 8B original-vs-quantised size table.
2. llama3.1:8b model page — Ollama. Lists the llama3.1:8b default tag as 4.9GB at Q4_K_M quantisation.