Quantisation and model selection

By the end, you can

• Explain what quantisation is and what GGUF's K-quant naming communicates at the level needed to choose a specific model file (LA-7).
• Weigh a real quantisation ladder's size, precision and speed trade-offs honestly, citing only what the source documentation itself states (LA-7).
• Choose between an instruct and a base model for a stated job, and trade off parameter count against quantisation level at a fixed memory budget (LA-7).

Before you start

This is Lesson 2 of 4 in Running models locally, building directly on Lesson 1's hardware sizing — that lesson established that a model's file size is a working stand-in for the memory it needs; this lesson explains where that file size actually comes from, and how to read the string in a model's filename or tag well enough to choose one deliberately rather than by guesswork. No installation is required yet — the setup lab is next.

What quantisation is, and how GGUF's K-quant naming encodes it

Quantisation is a precision reduction. llama.cpp's own documentation states it directly: "Quantization reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference. This process however, may introduce some accuracy loss." A model trained and stored at high precision (commonly 16- or 32-bit numbers per weight) is converted to a lower-precision representation — fewer bits per weight — producing a smaller file that is faster to load and run, at the cost of some accuracy that the source is candid is real but does not itself quantify for every case.

GGUF is the file format this quantised model gets packaged into for local runtimes. Hugging Face's own documentation describes it as "a binary format that is optimized for quick loading and saving of models, making it highly efficient for inference purposes," "designed for use with GGML and other executors," and notes it "was developed by @ggerganov who is also the developer of llama.cpp." A GGUF file's name commonly ends in a quantisation code such as `Q4_K_M` — and that code is not decorative. Breaking it down against Hugging Face's own Quantization Types table:

• The leading digit is the nominal bits-per-weight family — `Q4` targets roughly 4-bit precision, though the actual figure is rarely a round number. The table's own worked definition for Q4_K states: "4-bit quantization (q). Super-blocks with 8 blocks, each block has 32 weights... resulting in 4.5 bits-per-weight" — genuinely 4.5, not a flat 4.
• The `K` marks a k-quant — llama.cpp's own `--pure` option is documented as disabling exactly this: "disable k-quant mixtures and quantizes all tensors to the same type." In other words, a K-quant does not apply one uniform bit-width across the whole model; it mixes more than one internal precision level across a model's different tensors by default, generally spending more bits where it matters more for output quality. A non-K "legacy" quant such as `Q8_0` or `Q4_0` applies one uniform level throughout, and Hugging Face's own table marks these legacy round-to-nearest types as "not used widely as of today" — though, as this lesson's worked example shows, "less common" is not the same as "never appropriate."
• A trailing `_S`, `_M` or `_L` (small, medium, large) names a size variant within the same bit-family, trading a little more size for a little more retained precision. This is not a naming convention taken on faith — Ollama's own qwen2.5 tag listing shows it as a real, checkable size progression on one live model: `qwen2.5:14b-instruct-q3_K_S` downloads at 6.7GB, `q3_K_M` at 7.3GB, `q3_K_L` at 7.9GB — same model, same nominal 3-bit family, growing file size as the suffix moves S → M → L.

Quality, size and speed move together — an honest trade-off

llama.cpp's own documentation gives a direct, worked comparison for one real model, Llama-3.1-8B, across its supported quant types. A sample of the table: `Q4_K_S` at 4.36 GiB, `Q4_K_M` at 4.58 GiB, `Q5_K_S` at 5.21 GiB, `Q5_K_M` at 5.33 GiB, `Q6_K` at 6.14 GiB, `Q8_0` at 7.95 GiB, and the unquantised `F16` at 14.96 GiB — with each type's exact bits-per-weight rising alongside its file size. The same table also records prompt-processing and text-generation throughput (tokens per second) for each type on the benchmark machine the document used — but the document does not state what that machine was, so this lesson treats those specific speed numbers as evidence that size and speed move together across the ladder on some real hardware, not as a claim about how fast any of these will run on your machine. Your own numbers, captured with a real runtime in Lesson 4, are the ones that matter for your hardware.

What the source documentation is candid about, and what this lesson will not overstate beyond it: quantising loses some accuracy, "usually measured in Perplexity (ppl)," and — reasonably — a lower-bit quant in a family generally has less headroom to preserve the original model's behavior than a higher-bit one in the same family. Neither cited source gives a specific perplexity number attached to a specific quant level for a specific model, so this lesson does not invent one; if a precise accuracy figure matters for your job, look it up against the specific model's own card or published evaluations before committing to a quant level, rather than assuming a rule of thumb from this lesson covers it.

A practical convention worth noticing rather than treating as a hard rule: `Q4_K_M` shows up repeatedly as a default tag across Ollama's library — it is the default for `llama3.2:3b`, `llama3.1:8b` and `qwen2.5:7b` (Lesson 1's worked figures) alike. That repeated choice is evidence of a widely-used practical balance point for mid-size instruct models, not a claim that it is the objectively correct choice for every job — a builder with tighter memory or a builder needing maximum retained quality both have legitimate reasons to choose differently, as the next section covers.

Choosing a model for the job: instruct vs base, and params vs quant at a fixed budget

Two separate choices sit underneath "which file do I download," and conflating them is a common mistake.

The first is instruct versus base. Meta's own model card for Llama 3.1 states the distinction plainly: "Instruction tuned text only models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks." For the bounded, chat-or-tool-shaped agent work this course builds toward, an instruct-tuned model is almost always the right starting point — it has already been trained to follow an instruction and hold up its end of a conversation. A base (pretrained) model is a different tool, suited to being adapted or fine-tuned for a specific generation task rather than run directly as a conversational agent; reach for one deliberately, not by default.

The second choice, once instruct-vs-base is settled, is how to spend a fixed memory budget: more parameters at a lower quantisation level, or fewer parameters at a higher one. Both routes can land at a similar file size, but they are not the same model in any meaningful sense — one has more raw capacity spent at lower per-weight precision, the other has less raw capacity spent at higher precision. Neither cited source in this course states a universal winner between these two shapes of trade-off; treat the choice as job-dependent, and where it matters, verify with your own evidence (Lesson 4) rather than a rule of thumb.

A worked example: two ways to spend the same 5-6 GB budget

Two real, currently-listed Ollama tags land in roughly the same disk-size range for very different reasons: `qwen2.5:7b`, at its default `Q4_K_M` quant, downloads at 4.7GB — a smaller parameter count held at a comparatively high per-weight precision. `qwen2.5:14b-instruct-q2_K`, the same model family at twice the parameter count but a much more aggressive 2-bit quant, downloads at 5.8GB — roughly the same disk footprint, spent very differently: double the raw parameter capacity, at meaningfully lower per-weight precision than the first option's Q4_K_M.

Neither this lesson nor its sources declare a winner between these two for every job — genuinely, it depends on what the job needs. A task that leans on broad world knowledge or handling more varied inputs might benefit from the larger parameter count even at lower precision; a task that leans on precise, consistent instruction-following at a smaller scope might do better with the smaller model held at higher precision. What this lesson wants you to leave with is not a rule that picks one for you, but the fact that this is a real, nameable trade-off — and that both options are checkable today, by the same size-lookup habit this module has used throughout, before you commit disk space and download time to either one.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. Model tag names and quantisation codes appear as plain inline text, read by a screen reader as ordinary text content; no information in this lesson depends on color or visual table layout alone. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. quantize tool README — ggml-org/llama.cpp. Defines quantisation as precision reduction, documents the --pure flag (k-quant mixtures vs uniform quantisation), and gives a Llama-3.1-8B size/bits-per-weight/speed comparison table across quant types.
2. GGUF — Hugging Face. Describes the GGUF file format and its Quantization Types table, defining Q4_K, Q5_K, Q6_K and marking legacy round-to-nearest types (Q4_0, Q5_0, Q8_0) as not widely used today.
3. qwen2.5 tag listing — Ollama. Lists real download sizes for the same qwen2.5:14b-instruct model across quant levels (Q2_K through Q3_K_S/M/L), showing the S/M/L suffix as a real size progression within one quant family.
4. qwen2.5:7b model page — Ollama. Lists the qwen2.5:7b default tag as 4.7GB at Q4_K_M, the smaller-parameter half of the lesson's fixed-budget worked example.
5. Llama 3.1 Model Card — Meta. States that instruction-tuned models are intended for assistant-like chat while pretrained (base) models are intended to be adapted for other generation tasks.
6. llama3.2 tag listing — Ollama. Shows llama3.2:1b defaulting to Q8_0 (1.3GB) despite Q8_0 being a legacy (non-K) format, illustrating that a smaller model can afford a higher default precision than a larger one.