What quantisation actually does to your model

What "quantisation" actually means

If you've browsed local model downloads, you've seen a file name like `llama-3.1-8b-instruct-Q4_K_M.gguf` and wondered what the letters after the parameter count are doing. The short answer is precision. A model's weights — the numbers that do the actual work of turning an input into an output — are usually trained and originally stored as 16- or 32-bit floating-point numbers. The tool that converts a model into a smaller download does something specific to those numbers, and its own documentation states it plainly: "Quantization reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference." Nothing about the model's structure changes — the same weights are still there, doing the same job — each one is just stored using fewer bits.

Why that matters before you've even run anything

Fewer bits per weight means a smaller file on disk, and — this is the part that actually decides whether a model runs on your machine at all — it also means less memory needed to load it. The same documentation is direct about this link: "As the models are currently fully loaded into memory, you will need adequate disk space to save them and sufficient RAM to load them. At the moment, memory and disk requirements are the same." In practice, that means a model's download size is a genuinely useful stand-in for the RAM (or VRAM, on a GPU) it will need — not a rough guess, but the actual figure you should size your hardware against.

Reading a name like Q4_K_M

GGUF is the file format most local runtimes — llama.cpp, Ollama, LM Studio — actually load. Hugging Face's documentation describes it as "a binary format that is optimized for quick loading and saving of models, making it highly efficient for inference purposes," and as "designed for use with GGML and other executors" — and notes it "was developed by @ggerganov who is also the developer of llama.cpp," which is why the format and that runtime fit together so naturally. A GGUF file's name commonly ends in a code like `Q4_K_M`, and each part of that code means something specific, per Hugging Face's own Quantization Types table:

Older, non-K types exist too — `Q4_0`, `Q5_0`, `Q8_0` — and apply one uniform precision level throughout rather than mixing. Hugging Face's own table marks each of these as a "Legacy quantization method (not used widely as of today)," so seeing one in a download listing today is the exception, not the default choice.

• The leading number is roughly the number of bits used per weight. `Q4_K` works out to 4.5 bits-per-weight in the table's own figures; `Q5_K` to 5.5; `Q6_K` to 6.5625.
• The `K` marks a k-quant — a mixed-precision scheme, not one uniform bit-width applied everywhere. You can confirm this by contrast: the quantisation tool's `--pure` option is documented as doing the opposite, to "disable k-quant mixtures and quantizes all tensors to the same type." That a flag exists specifically to switch mixing off tells you mixing is the K-quant default.
• A trailing `S`, `M` or `L` (small/medium/large) marks a size variant within the same bit-family — a real, checkable difference in file size and precision, not a cosmetic label.

What quantisation actually costs you

None of this is free. The same documentation that defines quantisation is candid about the trade-off: "This process however, may introduce some accuracy loss which is usually measured in Perplexity (ppl) and/or Kullback–Leibler Divergence (kld)." Neither source cited by this article gives one universal number for how much accuracy a specific quant level costs — that depends on the specific model and the specific quant — so this article won't invent a rule of thumb where the primary sources don't give one. What both sources do support is a general, honestly-hedged shape: lower-bit quants in a family generally carry more risk of losing some of the original model's behaviour than higher-bit quants in the same family, and if a precise figure matters for your use case, it's worth checking the specific model's own card or published evaluations rather than assuming.

A worked example: one model, several files

The quantisation tool's own documentation gives a real, worked comparison for one model — Llama-3.1-8B — across several quant types. A sample of its table, by size and bits-per-weight: `Q4_K_S` at 4.36 GiB (4.6672 bits/weight), `Q4_K_M` at 4.58 GiB (4.8944 bits/weight), `Q5_K_S` at 5.21 GiB (5.5704 bits/weight), `Q5_K_M` at 5.33 GiB (5.7036 bits/weight), `Q6_K` at 6.14 GiB (6.5633 bits/weight), `Q8_0` at 7.95 GiB (8.5008 bits/weight), and the unquantised `F16` at 14.96 GiB (16.0005 bits/weight). Read across that table and the pattern is exactly what the earlier sections predicted: file size and bits-per-weight rise together, and the gap between the smallest quantised option and the unquantised original is large — roughly a third of the size at `Q4_K_M`, for the same 8-billion-parameter model. The same table also records the document's own prompt-processing and generation-speed benchmarks for each type, but since the document doesn't state what hardware produced those numbers, treat them as evidence that size and speed move together on some real machine, not as a promise about how fast any of these will run on yours.

The one thing to remember

Quantisation is a size-for-precision trade, applied per weight, with the file's download size doubling as a genuinely reliable estimate of the memory it needs to run. A `K`-suffixed quant mixes precision levels across the model rather than applying one flat level, and the `S`/`M`/`L` suffix is a real size step within that mixture, not decoration. Everything else — which specific quant is "worth it" for a specific job on specific hardware — is a judgement call this article deliberately leaves to you, backed by the primary documentation rather than a guessed rule.

If you want the hands-on version of this — sizing a real model against your own machine's RAM, choosing between an instruct and a base model for a specific task, and measuring your own latency rather than trusting someone else's benchmark — Learning Harbour's Local AI Agents course covers it in depth in its "Running models locally" module.

Sources and limits

This article synthesises the sources below into a practical explanation. It is not a security standard, legal advice, or a guarantee that guidance current at review time still applies — check the review date above against your own situation.

1. quantize tool README — ggml-org/llama.cpp. Defines quantisation, documents the --pure flag, states memory and disk requirements are currently the same for a fully-loaded model, and gives a worked Llama-3.1-8B size/bits-per-weight table across quant types.
2. GGUF — Hugging Face. Describes the GGUF file format and its Quantization Types table, including Q4_K/Q5_K/Q6_K definitions and which legacy types are marked not widely used today.