Quantization formats explained
GGUF, AWQ, GPTQ, EXL2 and MLX — what they are, and what Q4_K_M means.
4 min read Reviewed July 2026
On this page
TL;DR
Pick the format your runtime reads: GGUF for llama.cpp and its ecosystem (LM Studio, Ollama — CPU, Mac, or a CPU+GPU split), AWQ/GPTQ/EXL2 for NVIDIA-GPU runtimes such as vLLM and ExLlamaV2, and MLX for Apple-native tooling on M-series Macs. Within GGUF, Q4_K_M is the standard starting point.
What does quantization actually do?
Quantization shrinks a model by storing its weights at lower numerical precision. A model trained in 16-bit (FP16/BF16) can be re-packed to roughly 8, 5, 4, or even 3 bits per weight. Fewer bits means a smaller file and less memory to load it, at the cost of some accuracy — the model's behaviour drifts a little further from the original with every bit you drop.
Two separate questions hide in "which quant do I download?": the format (decided by your software) and the bit-rate (decided by your memory — see the VRAM guide). The formats below differ mainly in how they pack the bits and which runtimes read them.
Which format does which runtime read?
| Format | Runtimes that read it | Hardware sweet spot | Typical bit-rates |
|---|---|---|---|
| GGUF | llama.cpp, LM Studio, Ollama, many desktop apps | CPU, Mac, or CPU+GPU split — the most portable | Wide range, roughly 2–8 bit, one file per level |
| AWQ | GPU runtimes such as vLLM | NVIDIA GPUs (full offload) | 4-bit |
| GPTQ | GPU runtimes such as vLLM | NVIDIA GPUs (full offload) | Mostly 4-bit; sometimes 3- or 8-bit |
| EXL2 | ExLlamaV2 | NVIDIA GPUs | Mixed bit-rates within one model |
| MLX | Apple's MLX tooling | Apple Silicon (M-series) unified memory | Various |
A little more on each:
- GGUF files are self-contained — weights, tokenizer, and metadata live in one file — and run on CPU, GPU, or a split of both. That portability is why it dominates local use (local LLM runtimes).
- AWQ (Activation-aware Weight Quantization) is a 4-bit scheme that protects the weights most sensitive to error.
- GPTQ is a widely used post-training method, also common on GPU runtimes.
- EXL2 supports mixed bit-rates within one model and is tuned for NVIDIA GPUs.
- MLX is Apple's array framework format, built for M-series machines and their unified memory (which Mac chip for LLMs?).
What does "Q4_K_M" mean?
GGUF quant names look cryptic but follow a pattern. Decoded piece by piece:
| Part | Meaning |
|---|---|
Q4 |
Roughly 4 bits per weight — the headline size/precision trade |
_K |
A K-quant: allocates bits more cleverly across blocks of weights than the older "legacy" quants |
_M |
The size class within that scheme — S (small), M (medium), L (large). A larger class spends a few more bits on the most important tensors |
So Q4_K_M is a 4-bit K-quant, medium size — a common default that balances file size against how close the output stays to the full-precision model. Q5_K_M and Q6_K keep more precision at a larger size; Q8_0 is near-lossless but roughly twice the size of a 4-bit quant. The full K-quant vs I-quant story, including the IQ names and imatrix builds, is in K-quants, I-quants and imatrix.
How do I choose?
Pick the format your runtime supports first, then the bit-rate your memory allows:
- Running on CPU, a Mac, or a mix of CPU+GPU → GGUF.
- Running fully on an NVIDIA GPU with vLLM/ExLlama → AWQ, GPTQ, or EXL2.
- On Apple Silicon and using MLX tooling → MLX.
As a rule of thumb, lower bit-rates diverge more from the original model, and the difference is usually more noticeable on smaller models than on large ones. If a quant fits comfortably, a higher bit-rate (Q5/Q6/Q8) preserves more of the original behaviour; if memory is tight, a 4-bit quant is the usual starting point. For a step-by-step ladder, see Which quant should I pick?
Is Q8_0 lossless?
No — near-lossless. FP16/BF16 is the reference; Q8_0 output stays very close to it but is not identical, at roughly twice the size of a 4-bit quant.
Can my GPU run GGUF?
Yes. GGUF is not CPU-only — llama.cpp-family runtimes can offload some or all layers to the GPU, and split the rest to CPU RAM. If the whole model fits on an NVIDIA card and you are running a server stack like vLLM or ExLlamaV2, the GPU-native formats (AWQ/GPTQ/EXL2) are built for exactly that path.
Do lower bit-rates hurt every model equally?
No. As a rule of thumb, quantization loss shows up more on smaller models than on large ones — a heavily quantized 7B degrades more noticeably than a 70B at the same level. Give small models more bits when you can.