LLM VRAM Calculator — Can I Run This LLM Locally?
Work out how much GPU memory a local LLM needs for your quantization and context length, whether it fits your GPU or Mac, and how fast it will run.
8.03B parameters. Attention: 32 layers x 8 KV heads x 128 dim.
Dark: weights · Blue: KV cache · Grey: overhead. GB here means GiB (1024³ bytes), the unit GPU memory is sold in.
| GPU / Mac | Usable | Verdict | Est. speed |
|---|---|---|---|
| RTX 3060 12 GB 360 GB/s | 12 GB | Fits | ~39 tok/s |
| RTX 4060 Ti 16 GB 288 GB/s | 16 GB | Fits | ~31 tok/s |
| RTX 4070 Super 12 GB 504 GB/s | 12 GB | Fits | ~55 tok/s |
| RTX 4080 Super 16 GB 736 GB/s | 16 GB | Fits | ~80 tok/s |
| RTX 5080 16 GB 960 GB/s | 16 GB | Fits | ~104 tok/s |
| RTX 3090 24 GB 936 GB/s | 24 GB | Fits | ~102 tok/s |
| RTX 4090 24 GB 1008 GB/s | 24 GB | Fits | ~110 tok/s |
| RTX 5090 32 GB 1792 GB/s | 32 GB | Fits | ~195 tok/s |
| Radeon RX 7900 XTX 24 GB 960 GB/s | 24 GB | Fits | ~104 tok/s |
| RTX 6000 Ada 48 GB 960 GB/s | 48 GB | Fits | ~104 tok/s |
| A100 40 GB 1555 GB/s | 40 GB | Fits | ~169 tok/s |
| A100 80 GB (SXM) 2039 GB/s | 80 GB | Fits | ~222 tok/s |
| H100 80 GB (SXM) 3350 GB/s | 80 GB | Fits | ~364 tok/s |
| H200 141 GB 4800 GB/s | 141 GB | Fits | ~522 tok/s |
| Ryzen AI Max+ 395 (96 GB to GPU) 256 GB/s | 96 GB | Fits | ~28 tok/s |
| Mac M2/M3/M4, 16 GB 100 GB/s | 10.7 GB | Fits | ~11 tok/s |
| Mac M3/M4, 24 GB 120 GB/s | 16.1 GB | Fits | ~13 tok/s |
| Mac M4 Pro, 48 GB 273 GB/s | 36 GB | Fits | ~30 tok/s |
| Mac M4 Max, 64 GB 546 GB/s | 48 GB | Fits | ~59 tok/s |
| Mac M4 Max, 128 GB 546 GB/s | 96 GB | Fits | ~59 tok/s |
| Mac M2 Ultra, 192 GB 800 GB/s | 144 GB | Fits | ~87 tok/s |
| Mac M3 Ultra, 512 GB 819 GB/s | 384 GB | Fits | ~89 tok/s |
"Tight" means within 10% of the limit: it may load, but a browser or another app can push it over. Speed is single-user generation, estimated as 65% of memory bandwidth divided by the bytes read per token; prompt processing is faster, and real numbers vary by runtime. Apple Silicon can give the GPU about two-thirds (up to 36 GB) or three-quarters (larger) of RAM by default.
Calculated in your browser. Nothing you enter is sent anywhere. Browse all free tools.
Found a problem, or something that could be better? We read every message and we fix things quickly.
This LLM VRAM calculator answers the question "can I run this LLM locally?" by estimating the memory a model needs on your GPU or Mac: the weights at your chosen quantization, the KV cache for your context length, and the runtime overhead. Pick a popular open model or enter a parameter count, choose a quantization such as Q4_K_M, and see the total next to a fits, tight or won't-fit verdict for common graphics cards and Apple Silicon machines, with a rough generation speed for each.
Most quick estimates multiply parameters by bytes per weight and stop there, which is close for short prompts and badly wrong for long ones. A Llama 3.1 8B model at Q4_K_M needs about 4.6 GB for weights, but its KV cache grows from 1 GB at an 8K context to 16 GB at 128K. The presets here carry each model's real attention layout (grouped-query heads, sliding windows, DeepSeek's latent attention), so the KV cache is sized the way llama.cpp and vLLM actually allocate it. The formulas are written out below and the maths is unit tested against real GGUF file sizes.
How to use
- Choose a model from the list, or Custom to enter a parameter count (and, for an exact KV cache, the layer, KV-head and head-dim values from the model's config.json).
- Pick the quantization you plan to download. Q4_K_M is the usual starting point for GGUF models in llama.cpp, Ollama and LM Studio.
- Set the context length you will actually use and the number of parallel sequences (1 for personal chat, more for a server).
- Read the total and the breakdown, then the verdict for each GPU or Mac. Tight means within 10% of the limit.
- If it does not fit, try a smaller quantization, a shorter context, a Q8_0 KV cache, or a model with fewer parameters.
Frequently asked questions
How much VRAM do I need to run an 8B model?
About 6.5 GB for Llama 3.1 8B at Q4_K_M with an 8K context: 4.6 GB of weights, 1 GB of KV cache and under 1 GB of overhead. That fits an 8 GB GPU and comfortably fits a 12 GB RTX 3060. At FP16 the same model needs about 17 GB, which means a 24 GB card. Context length matters: at the full 128K context the KV cache alone is 16 GB.
Can I run a 70B model on a 24 GB GPU?
Not entirely in VRAM. Llama 3.3 70B at Q4_K_M needs about 40 GB for weights and roughly 45 GB in total at an 8K context; even Q2_K needs about 30 GB. Options are two 24 GB GPUs, one 48 GB card, a Mac with 64 GB or more of unified memory, or offloading part of the model to system RAM, which works but is several times slower.
What is the KV cache and why does it grow with context?
During generation the model stores a key and a value vector for every previous token in every layer, so it does not recompute them for each new token. Its size is 2 x layers x KV heads x head dimension x context length x bytes per value. For Llama 3.1 8B (32 layers, 8 KV heads, head dim 128, FP16) that is 128 KB per token: 1 GB at 8K tokens and 16 GB at 128K. Models with grouped-query attention, sliding-window layers (Gemma) or latent attention (DeepSeek) need far less.
Which quantization should I use?
Q4_K_M is the standard trade-off: about a third of the FP16 size with a small quality loss for most uses. Use Q5_K_M or Q6_K if they fit, Q8_0 when you want results practically identical to the original, and avoid Q3 and Q2 on models under about 30B parameters, where the quality drop becomes obvious. A larger model at Q4 usually beats a smaller model at Q8 of the same file size.
Is a Mac or an NVIDIA GPU better for local LLMs?
A Mac fits bigger models; an NVIDIA GPU runs the ones that fit faster. Apple Silicon shares one pool of memory between CPU and GPU, so a 128 GB Mac can hold a 70B model that no consumer graphics card can, but its memory bandwidth (546 GB/s on an M4 Max) is about half an RTX 4090's 1,008 GB/s, and generation speed scales with bandwidth. By default macOS lets the GPU use about two-thirds of RAM on machines up to 36 GB and about three-quarters on larger ones.
How accurate is the tokens-per-second estimate?
It is a rough guide, typically within 30% for single-user generation. Generating each token means reading the active weights and the KV cache from memory once, so speed is about memory bandwidth divided by bytes per token; the tool assumes 65% of peak bandwidth is achieved. Prompt processing is compute-bound and much faster, MoE models add routing overhead, and runtime, drivers and settings all shift real numbers.
The formulas this calculator uses
Total memory = model weights + KV cache + runtime overhead. All sizes are in GiB (1,024³ bytes), which is the unit GPU memory is sold in: a "24 GB" RTX 4090 has 24 GiB.
Weights = parameters x bits per weight / 8. For GGUF k-quants the bits per weight are effective values measured from real files, because each quant type mixes precisions across tensors: Q8_0 8.5, Q6_K 6.57, Q5_K_M 5.70, Q4_K_M 4.89, IQ4_XS 4.43, Q3_K_M 4.00, Q2_K 3.17 (from the Llama 3.1 8B GGUF files; larger models come out up to about 5% smaller). AWQ and GPTQ use 4-bit weights with one scale per 128 weights (about 4.16 bits) and keep the embeddings and output layer at 16-bit; gpt-oss ships its expert weights in MXFP4 (4.25 bits) with attention and embeddings in BF16.
KV cache = 2 x layers x KV heads x head dim x tokens x bytes per value x parallel sequences. Grouped-query attention is why kv_heads (8 for Llama 3 70B) is much smaller than the number of attention heads (64). Sliding-window layers store at most their window (1,024 tokens on Gemma 3, 128 on half of gpt-oss's layers). DeepSeek V3 and R1 store one 576-value latent per layer per token instead of full keys and values. FP16 uses 2 bytes per value, a Q8_0 cache about 1.06 and Q4_0 about 0.56.
Overhead = 0.75 GiB + 3% of the weights, covering the CUDA or Metal context and compute buffers. Real runtimes differ by a few hundred megabytes. Speed = memory bandwidth x 0.65 / (active weight bytes + one sequence's KV cache).
Quantization sizes at a glance
Weights only, from this calculator, for Llama 3.1 8B and Llama 3.3 70B:
• FP16 / BF16: 15.0 GB and 131.5 GB
• Q8_0: 7.9 GB and 69.9 GB
• Q6_K: 6.1 GB and 54.0 GB
• Q5_K_M: 5.3 GB and 46.8 GB
• Q4_K_M: 4.6 GB and 40.2 GB
• Q3_K_M: 3.7 GB and 32.9 GB
• Q2_K: 3.0 GB and 26.1 GB
Add the KV cache for your context and about 1 GB of overhead to get what the GPU must hold.
Mixture-of-experts models: memory of a big model, speed of a small one
A mixture-of-experts (MoE) model such as Qwen3 30B-A3B, Mixtral 8x7B, gpt-oss or DeepSeek R1 contains many expert sub-networks but uses only a few for each token. All of them must be in memory, so memory follows the total parameter count. Speed follows the active parameters, because only those weights are read per token.
That is why Qwen3 30B-A3B at Q4_K_M needs about 19 GB, the memory of a dense 30B model, but generates text at the speed of a 3B one. It is also why MoE models suit Macs and other machines with plenty of slower memory: capacity is the constraint, bandwidth less so.
DeepSeek R1 at 671B parameters is the extreme case: about 380 GB at Q4_K_M, beyond any single consumer machine, even though only 37B parameters are active per token.
Related free tools
- LLM API Cost Calculator — Compare the cost of your workload across 40+ models, with caching, batch discounts, long-context tiers and reasoning tokens. Prices sourced and dated.
- Token Counter — Count tokens exactly for OpenAI models, estimate them for Claude, Gemini and Llama, see the cost, and watch the text split into tokens.
- AI ROI Calculator — Estimate how many hours and dollars automating a repetitive task could save your business every year.