How to calculate the memory (VRAM) a local LLM needs

1. Model file size Measured

llama.cpp-based tools load the GGUF file into memory as is, so the memory used by the model weights is almost the same as the file size. This site uses the file size returned by Hugging Face (the actual value, in bytes). If the file is split, as in -00001-of-00003.gguf, we use the total of all parts.

2. KV cache Estimated

Each time an LLM generates a token, it refers to the "keys" (K) and "values" (V) of all previous tokens. The KV cache stores them so they do not have to be recomputed every time, and it grows in proportion to the context length (the number of tokens handled).

The values in the formula come from the original model's config.json

For Qwen3-8B, each token takes 2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes (144 KiB). At 32K tokens (32,768) that is 4,831,838,208 bytes = 4.83 GB (exactly 4.5 GiB). Source: config.json of Qwen/Qwen3-8B (checked 2026-09-26)

3. Sliding window layers only need the window

Gemma 3 and gpt-oss have sliding window attention (SWA) layers that only look at a fixed number of recent tokens (the window). For these layers, llama.cpp allocates the KV cache not for the context length but only for "window + 512", rounded up to a multiple of 256 (512 is the default n_ubatch, the number of tokens processed at once).

Gemma 3 27B (Q4_K_M): how much accounting for SWA changes the result

gpt-oss-20b has a small window of 128, so even at its maximum of 131,072 tokens the KV cache is 3.24 GB (of which the SWA layers take 19 MB), and the estimated memory requirement is 15.8 GB (Q4_K_M, 11.6 GB).

4. KV cache quantization (q8_0, q4_0)

llama.cpp can shrink the KV cache itself by quantizing it (--cache-type-k q8_0 --cache-type-v q8_0). Quantized types store 32 elements together as one block, so the bytes per element are as follows.

Quantizing the V cache requires Flash Attention (--flash-attn on). The change in accuracy from quantization differs between models and this site has not measured it, so the tables use f16 as the default and list q8_0 and q4_0 for reference.

5. Compute buffer and reserve Estimated

6. Quick reference by context length (Qwen3-8B Q4_K_M)

Estimated memory requirements for a model with a file size of 5.03 GB Measured , as the context length and KV cache type change. Each model's page also has a table calculated with the same formula.

All values are Estimated .

7. Assumptions, and when they do not hold

These estimates assume llama.cpp (b11639, as of October 2026) running with the settings below, which are the defaults of llama-server and llama-cli.

• 1 All layers are offloaded to the GPU (unified memory on a Mac) Keeping some layers on the CPU side reduces what the GPU needs, at the cost of speed.
• 2 Only one conversation is processed at a time Increasing the number of parallel sequences (-np) in llama-server increases the KV cache of the SWA layers with the number of sequences.
• 3 n_ubatch = 512 (default), Flash Attention enabled With Flash Attention turned off, the compute buffer grows considerably at long context lengths.
• 4 The context length is the value given in each row of the table Ollama and LM Studio decide the context length in their own settings. Check the length you use in the settings and look at the row for that value.

Cross-checking against llama.cpp

We have confirmed that the KV cache and the fixed-size state held by layers such as linear attention match, to the byte, what llama.cpp itself actually allocates. Here is how we checked. We converted the config.json of each listed model into only the settings part of a GGUF using llama.cpp's official conversion code, had llama.cpp create a context without loading any weights, and read out the amount it allocated. We tested four context lengths: 4,096, 8,192, 32,768 and 131,072 tokens.

• • Version used: llama.cpp b11639 (October 14, 2026). We will check again when the version changes.
• • Not verified: the compute buffer and reserve (the values in section 5). These remain estimates until we measure them by actually running on a GPU.
• • SWA layers are assumed to allocate only the window, as llama-server and llama-cli do by default (with --swa-full the full context is allocated).

Models that hold the KV cache differently

For the following models, we checked llama.cpp's implementation (how it allocates the KV cache) and then applied the same formula layer by layer.

Models combining linear attention (Qwen3.5, Qwen3-Next, etc.)

Only the regular attention layers hold a KV cache (usually one layer in four). The remaining linear attention layers hold a state of fixed size that does not depend on the length of the conversation. The state size per layer is ((convolution width − 1) × (d_inner + 2 × K heads × K head dimension) + K head dimension × d_inner) × 4 bytes, and we add it to the estimated memory requirement (d_inner = V heads × V head dimension). A key feature is that even for long conversations the KV cache is about 1/4 that of a regular model.

Linear attention + sparse attention (Qwen3.8-Flash-Next, etc.)

In addition to the linear attention model above, each regular attention layer holds an indexer cache (1 head × indexer_head_dim, K only) used to choose which tokens to attend to, with the same length as the KV cache. It also holds a fixed state of (ple_conv_kernel_size − 1) × ngram_size × hc_count × hidden_size × 4 bytes as the convolution history of the n-gram embeddings (PLE).

Models combining convolutions (LFM2 family)

Only the regular attention layers hold a KV cache; the convolution layers hold a fixed state of hidden_size × (conv_L_cache − 1) × 4 bytes.

Gemma 4

The SWA layers and the layers that keep the full context have different numbers of KV heads and different head_dim (num_global_key_value_heads and global_head_dim in config.json). We calculate the KV cache with the width of each layer. In E2B, E4B and similar variants, the last layers (num_kv_shared_layers) reuse the KV cache of earlier layers, so nothing is allocated for them. Likewise, in Gemma 3n the last layers do not allocate a KV cache.

DeepSeek V4

It has a dedicated KV cache. In addition to the KV (K only) for the most recent 128 tokens, some layers hold K compressed 4 tokens into 1 plus an indexer K used to choose where to read from, and other layers hold K compressed 128 tokens into 1. We also add the fixed state used by the compression mechanism. As a result, the KV cache stays small even for long conversations, around 1 GB even at 131,072 tokens (most of the memory requirement is the model file itself).

Models combining Mamba2 (Nemotron-H family)

Only the attention layers hold a KV cache; the Mamba2 layers hold a fixed state of ((conv_kernel − 1) × (d_inner + 2 × n_groups × ssm_state_size) + ssm_state_size × d_inner) × 4 bytes (d_inner = mamba_num_heads × mamba_head_dim).

Chunked attention (Llama 4)

Three out of every four layers only look within chunks of 8,192 tokens. llama.cpp allocates the same amount for these layers as for SWA (chunk length + 512), so we use the same formula.

Embedding models (EmbeddingGemma, etc.)

They read the text in one pass and turn it into numbers without generating a continuation, so they have no KV cache.

8. Models we do not estimate

For models that hold the KV cache differently from the formulas above, we do not show an estimate, to avoid showing wrong numbers (the reason is given on the model's page). The same applies when the original model's config.json cannot be retrieved (for example, models that require agreeing to terms before access). However, for models where we could read the settings at the start of the GGUF file (the very values llama.cpp reads), we estimate from those. For MLA models (DeepSeek V3, Kimi K2, GLM-5.3, etc.), we use the GGUF settings to identify when the model was converted (the shape of the KV) before estimating.

model_types we can estimate: llama, mistral, mixtral, qwen2, qwen2_moe, qwen3, qwen3_moe, phi3, qwen3_vl_text, granite, ministral3, hunyuan_v1_dense, qwen2_5_vl, qwen2_5_omni_text, qwen3_vl_moe_text, qwen3_omni_moe_text, glm4, glm4v_text, smollm3, hy_v3, k2_horizon, chatglm, nanbeige, gemma2, gemma3, gemma3_text, gpt_oss, muse_glimmer_text, spark2_5, laguna, cohere2_moe, mellum, afmoe, step3p5, llama4_text, qwen3_next, qwen3_5_text, qwen3_5_moe_text, qwen4_exp_text, gemma4_text, gemma4_unified_text, lfm2, lfm2_moe, embedding_gemma2_text, jina_embeddings_v5, gemma3n_text, mimo_v2, nemotron_h, deepseek_v4

FAQ