Guide
How to read and choose quantizations (Q4_K_M, Q8_0 and more)
GGUF files of local LLMs carry a "quantization type" such as Q4_K_M or Q8_0. The type changes the file size and how far the output drifts from the original model. This page explains how to read the names, with sizes aggregated from the real files listed on this site and a quality guide.
Aggregates updated daily (from 20,780 files across 1,276 models on this site)
Last updated
What quantization is
LLM weights (parameters) are originally stored with 16 bits each (BF16 or F16). Quantization re-expresses them with fewer bits, such as 8, 4 or 2, to make the file and the required memory smaller.
Fewer bits means a smaller file but a larger drift from the original model's output (lower quality). Required memory is roughly "file size + KV cache (grows with conversation length) + some headroom", so the starting point is finding a type that fits your VRAM.
How to read the names
A name is a combination of parts.
| Part | Meaning |
|---|---|
| Q4, Q5, Q8 etc. | Roughly how many bits per weight (Q4 is about 4 bits). Scales and other data add a little on top. |
| _K (Q4_K_M etc.) | k-quants: weights are split into small blocks, each with its own scale. Closer to the original than the older formats (_0, _1) at the same bits. |
| _S, _M, _L | Size within the same type (Small, Medium, Large). Larger ones spend more bits on important layers and stay closer to the original. |
| IQ (IQ4_XS, IQ2_M etc.) | i-quants: designed to keep drift small at low bit counts; common for 2–4 bit types and used with an imatrix. |
| _0, _1 (Q4_0, Q8_0 etc.) | Older, simple formats. Q8_0 uses 8 bits and gives nearly the same output as the original. |
| F16, BF16, F32 | Original precision, not quantized. Largest, with no quality drift. |
| MXFP4, NVFP4 | 4-bit floating-point formats. Some authors ship models in these formats from the start (gpt-oss, for example). |
| UD-, *_XL, *_L (distributor types) | Types mixed per layer by a distributor (such as Unsloth's UD-Q4_K_XL). The name alone does not define the contents; check the distributor's notes. |
| imatrix (i1 etc.) | Quantized using information about which weights matter most. Slightly closer to the original at the same type. |
Real sizes and quality guide by type
Bits per parameter computed by dividing the real size of the GGUF files listed on this site by the parameter count written in the GGUF (from 20,780 files). The same type varies by model and distributor.
| Type | Per parameter (median) Measured | Middle half | For an 8B model | Files | Quality guide |
|---|---|---|---|---|---|
| F32 | 32.01 bits | 32.01–32.04 | 32.0 GB | 44 | Original precision |
| BF16 | 16.01 bits | 16.00–16.02 | 16.0 GB | 385 | Original precision |
| F16 | 16.01 bits | 16.00–16.02 | 16.0 GB | 447 | Original precision |
| UD-Q8_K_XL | 9.52 bits | 9.09–10.57 | 9.5 GB | 107 | - |
| Q8_K | 9.51 bits | 9.07–10.06 | 9.5 GB | 11 | - |
| Q8_0 | 8.51 bits | 8.50–8.52 | 8.5 GB | 1,309 | Nearly identical |
| UD-Q6_K_XL | 7.34 bits | 7.18–7.63 | 7.3 GB | 109 | - |
| Q6_K_L | 6.90 bits | 6.77–7.25 | 6.9 GB | 126 | - |
| Q6_K_S | 6.71 bits | 6.69–6.73 | 6.7 GB | 12 | - |
| UD-Q6_K | 6.70 bits | 6.61–6.76 | 6.7 GB | 17 | - |
| Q6_K | 6.57 bits | 6.57–6.61 | 6.6 GB | 1,252 | Very small |
| Q5_1 | 6.03 bits | 6.02–6.05 | 6.0 GB | 122 | Small |
| Q5_K_L | 6.02 bits | 5.91–6.31 | 6.0 GB | 113 | - |
| UD-Q5_K_M | 6.02 bits | 5.94–6.11 | 6.0 GB | 16 | - |
| UD-Q5_K_XL | 5.97 bits | 5.75–6.12 | 6.0 GB | 105 | - |
| UD-Q5_K_S | 5.76 bits | 5.64–5.79 | 5.8 GB | 19 | - |
| Q5_K_M | 5.71 bits | 5.69–5.80 | 5.7 GB | 1,179 | Very small |
| Q5_K | 5.69 bits | 5.68–5.76 | 5.7 GB | 84 | - |
| Q5_K_S | 5.58 bits | 5.53–5.63 | 5.6 GB | 941 | Small |
| Q5_0 | 5.55 bits | 5.52–5.58 | 5.6 GB | 255 | Small |
| Q4_K_L | 5.31 bits | 5.13–5.54 | 5.3 GB | 131 | - |
| TQ3_0 | 5.26 bits | 3.48–6.89 | 5.3 GB | 6 | - |
| Q4_K_XL | 5.18 bits | 5.13–5.43 | 5.2 GB | 10 | - |
| Q4_1 | 5.10 bits | 5.04–5.18 | 5.1 GB | 390 | Medium |
| UD-Q4_K_M | 5.08 bits | 4.95–5.11 | 5.1 GB | 19 | - |
| UD-Q4_K_XL | 5.05 bits | 4.89–5.27 | 5.0 GB | 129 | - |
| Q4_K_M | 4.92 bits | 4.87–5.06 | 4.9 GB | 1,612 | Small |
| Q4_K | 4.86 bits | 4.84–4.99 | 4.9 GB | 98 | - |
| UD-Q4_K_S | 4.82 bits | 4.63–4.82 | 4.8 GB | 19 | - |
| MXFP4_MOE | 4.80 bits | 4.63–5.01 | 4.8 GB | 38 | - |
| NVFP4 | 4.76 bits | 4.65–4.96 | 4.8 GB | 17 | - |
| Q4_K_S | 4.67 bits | 4.59–4.77 | 4.7 GB | 968 | Medium |
| IQ4_NL | 4.66 bits | 4.57–4.78 | 4.7 GB | 448 | Small |
| Q3_K_XL | 4.65 bits | 4.36–4.86 | 4.6 GB | 114 | - |
| Q4_0_4_8 | 4.64 bits | 4.60–4.73 | 4.6 GB | 35 | - |
| Q4_0_8_8 | 4.64 bits | 4.59–4.74 | 4.6 GB | 33 | - |
| Q4_0_4_4 | 4.64 bits | 4.56–4.71 | 4.6 GB | 38 | - |
| Q4_0 | 4.63 bits | 4.55–4.72 | 4.6 GB | 677 | Medium |
| MXFP4 | 4.46 bits | 4.39–4.63 | 4.5 GB | 10 | - |
| IQ4_XS | 4.43 bits | 4.34–4.53 | 4.4 GB | 919 | Small |
| Q3_K_L | 4.29 bits | 4.22–4.37 | 4.3 GB | 847 | Medium |
| UD-IQ4_NL | 4.09 bits | 3.95–4.18 | 4.1 GB | 18 | - |
| UD-IQ4_XS | 4.09 bits | 3.88–4.21 | 4.1 GB | 33 | - |
| UD-Q3_K_XL | 4.06 bits | 3.83–4.36 | 4.1 GB | 117 | - |
| Q3_K_M | 4.00 bits | 3.89–4.12 | 4.0 GB | 1,014 | Medium |
| Q3_K | 3.92 bits | 3.89–4.00 | 3.9 GB | 82 | - |
| IQ3_M | 3.76 bits | 3.67–3.92 | 3.8 GB | 526 | Medium |
| UD-Q3_K_M | 3.73 bits | 3.62–3.85 | 3.7 GB | 15 | - |
| Q3_K_S | 3.63 bits | 3.51–3.73 | 3.6 GB | 868 | Large |
| IQ3_S | 3.60 bits | 3.51–3.68 | 3.6 GB | 352 | Medium |
| Q2_K_L | 3.50 bits | 3.16–3.80 | 3.5 GB | 163 | - |
| IQ3_XS | 3.49 bits | 3.34–3.61 | 3.5 GB | 469 | Large |
| UD-IQ3_XXS | 3.33 bits | 3.17–3.62 | 3.3 GB | 108 | - |
| UD-Q2_K_XL | 3.30 bits | 2.90–3.65 | 3.3 GB | 119 | - |
| UD-IQ3_S | 3.27 bits | 3.16–3.46 | 3.3 GB | 24 | - |
| IQ3_XXS | 3.26 bits | 3.12–3.46 | 3.3 GB | 377 | Large |
| Q2_K | 3.17 bits | 3.01–3.32 | 3.2 GB | 932 | Very large |
| UD-IQ2_M | 3.04 bits | 2.80–3.35 | 3.0 GB | 94 | - |
| Q2_K_S | 2.96 bits | 2.81–3.08 | 3.0 GB | 134 | Very large |
| IQ2_M | 2.94 bits | 2.76–3.11 | 2.9 GB | 392 | Large |
| TQ2_0 | 2.77 bits | 2.57–5.85 | 2.8 GB | 9 | - |
| Q2_0 | 2.73 bits | 2.27–3.01 | 2.7 GB | 17 | - |
| IQ2_S | 2.71 bits | 2.54–2.81 | 2.7 GB | 315 | Very large |
| UD-IQ2_XXS | 2.59 bits | 2.49–2.82 | 2.6 GB | 92 | - |
| IQ2_XS | 2.56 bits | 2.43–2.66 | 2.6 GB | 330 | Very large |
| UD-IQ1_M | 2.39 bits | 2.27–2.55 | 2.4 GB | 73 | - |
| TQ1_0 | 2.35 bits | 1.89–3.72 | 2.3 GB | 8 | - |
| IQ2_XXS | 2.35 bits | 2.20–2.51 | 2.3 GB | 291 | Very large |
| UD-IQ1_S | 2.27 bits | 2.15–2.41 | 2.3 GB | 61 | - |
| UD-TQ1_0 | 2.11 bits | 2.02–2.16 | 2.1 GB | 14 | - |
| IQ1_M | 2.10 bits | 1.92–2.20 | 2.1 GB | 265 | Very large |
| IQ1_S | 1.95 bits | 1.78–2.05 | 1.9 GB | 245 | Very large |
| Q1_0 | 1.23 bits | 1.14–1.43 | 1.2 GB | 12 | - |
"For an 8B model" is the file size implied by the median bits (8,000,000,000 parameters × bits ÷ 8). The quality guide is a level from values llama.cpp measured on Llama-3-8B (mean KLD), not a measurement of each model.
Steps to choose for your VRAM
- Check how much VRAM your GPU has (unified memory on a Mac).
- On the model's page, find the quantizations that fit that amount. Required memory grows with context length, so look at the column for the length you will use.
- Among those that fit, check the quality guide. Types smaller than around Q4_K_M drift further from the original.
- Both "a small model at a large type" and "a large model at a small type" are options. Use third-party evaluations (rankings) as a reference and test for your use.
Related pages
Frequently asked questions
- What is the difference between Q4_K_M and Q4_K_S?
- Both are roughly 4-bit k-quants, but Q4_K_M spends more bits on some important layers, so it is slightly larger and closer to the original.
- What is the difference between Q8_0 and F16 (BF16)?
- Q8_0 uses 8 bits, so the file is about half the size of F16. In llama.cpp's measurement, its drift from the original is very small (mean KLD 0.0014).
- What is different about imatrix (i1) files?
- They were quantized using an importance matrix (which weights matter most). At the same type and size, they stay slightly closer to the original.
- Is this different from KV cache quantization?
- Yes. This page is about quantizing the model weights (the file). The KV cache (memory that grows with conversation length) can also be quantized to q8_0 or q4_0 to reduce memory; see the memory calculation page.