Tools / LLM GPU calculator
How many GPUs does your model need?
VRAM for the weights plus KV cache, mapped to data center GPUs, RTX cards and Mac Studio. Every formula is on the page.
70.6B params · 80 layers · 8 KV heads · head dim 128
Weight precision
KV cache precision
Capped at 131,072 tokens, the model's max_position_embeddings.
Requests in flight at the same moment, not total staff.
Estimated GPU memory needed
306 GB
- Weights
- 141.2 GB
- KV cache
- 134.2 GB
- KV per token
- 0.328 MB
- Runtime headroom
- 10%
| GPU | Memory | GPUs by memory | Estimated memory fit |
|---|---|---|---|
| Data center · tensor parallel | |||
| NVIDIA L40S | 48 GB GDDR6 | 7 | 8 GPUs · TP8 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition | 96 GB GDDR7 | 4 | 4 GPUs · TP4 |
| NVIDIA H100 SXM | 80 GB HBM3 | 4 | 4 GPUs · TP4 |
| NVIDIA H100 NVL | 94 GB HBM3 | 4 | 4 GPUs · TP4 |
| NVIDIA H200 | 141 GB HBM3e | 3 | 4 GPUs · TP4 |
| NVIDIA B200 | 180 GB HBM3e | 2 | 2 GPUs · TP2 |
| NVIDIA B300 | 288 GB HBM3e | 2 | 2 GPUs · TP2 |
| AMD Instinct MI300X | 192 GB HBM3 | 2 | 2 GPUs · TP2 |
| AMD Instinct MI355X | 288 GB HBM3E | 2 | 2 GPUs · TP2 |
| Workstation & consumer · layer split | |||
| NVIDIA GeForce RTX 3090 | 24 GB GDDR6X | 13 | 13 cards (layer split, e.g. llama.cpp) |
| NVIDIA GeForce RTX 4090 | 24 GB GDDR6X | 13 | 13 cards (layer split, e.g. llama.cpp) |
| NVIDIA GeForce RTX 5090 | 32 GB GDDR7 | 10 | 10 cards (layer split, e.g. llama.cpp) |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition | 96 GB GDDR7 | 4 | 4 cards (layer split, e.g. llama.cpp) |
| Apple Mac Studio M3 Ultra / M5 Ultra | 96 GB unified memoryapprox. 75% usable for the GPU | 5 | 5 Macs (layer split, e.g. llama.cpp) |
| Apple Mac Studio M3 Ultra / M5 Ultra | 256 GB unified memoryapprox. 75% usable for the GPU | 2 | 2 Macs (layer split, e.g. llama.cpp) |
| Apple Mac Studio M5 Ultra | 512 GB unified memoryapprox. 75% usable for the GPU | 1 | 1 Mac |
Estimated memory fit: how many GPUs the model spreads across inside a compatible system (TP = tensor-parallel size), not a server to buy. Two B200s means two GPUs of an 8-GPU HGX B200 server, for example. Workstation and consumer cards are counted for a layer split, which works with any number of cards; Mac Studio counts about 75% of unified memory as usable by the GPU.
Memory is step one. Throughput, latency, power and the network decide the real build.
Size my actual workloadHow the calculator works
Weights = parameters × bytes per weight: 2 for FP16/BF16, 1 for FP8, about 0.55 for INT4 (0.5 plus group-wise scales).
KV cache = 2 (keys and values) × layers × KV heads × head dimension × bytes per value × context tokens × concurrent sequences. Models with grouped-query attention, like Llama 3, keep far fewer KV heads than attention heads, which is why their cache is small per token.
MLA (multi-head latent attention, in DeepSeek-V3/R1 and GLM-5) caches one compressed latent per layer instead of per-head keys and values: layers × (kv_lora_rank + qk_rope_head_dim) × bytes per token. DeepSeek-V3 needs about 0.07 MB per token at FP16, against 0.33 MB for Llama 3.3 70B.
Sliding-window attention (Gemma 3, gpt-oss, and Llama 4’s chunked attention) lets local layers look back only a fixed window, so they cache min(context, window) tokens while global layers cache the full context; the two groups are added separately. Layers without attention, like LFM2.5’s convolutions, cache nothing per token.
Required memory = (weights + KV cache) ÷ 0.9, leaving 10% for activations, CUDA graphs and fragmentation; vLLM’s default memory utilization is 0.9.
Deploy as rounds up to a tensor-parallel size the model can actually be split by: it must divide both the attention-head and KV-head counts (attention heads only for MLA), so 1, 2, 4 or 8 per server. If no such size up to eight holds the model, it spans multiple servers, which brings in the network. Workstation and consumer cards are counted for a layer split, as llama.cpp does, where any number of cards works.
Any Hugging Face model: type its ID (org/model) or paste its link, and the page reads total parameters from the Hub’s count of its safetensors files and the architecture from its config.json, straight from your browser. Gated repos (Llama, Gemma and others) only show config.json to signed-in users who accept their terms, so paste it in instead. Custom model takes the architecture from a pasted config.json or by hand; config.json doesn’t state parameter counts reliably, so enter those yourself. The context slider stops at each model’s max_position_embeddings, and every input is in the page URL, so you can link to an exact scenario.
This sizes capacity, not speed. Tokens per second depend on memory bandwidth, batch size, prompt-to-output ratio and the serving stack. Mixture-of-experts models count every expert (total parameters, not active), because all of them must be resident. GPU figures are vendor-specified per-GPU capacities; Mac Studio counts about 75% of unified memory, the share macOS lets the GPU use by default. Memory is local to each GPU; spreading a model across several needs tensor or pipeline parallelism in the serving software. Worked example: how many GPUs a 70B model needs for 100 concurrent users. Model figures updated .
VRAM requirements by model
One user at an 8K context, KV cache at FP16. Each page has GPU counts at every precision, the KV cache formula and the model’s config.json.
- Llama 3.1 8B19 GB FP16 · 6.1 GB 4-bit
- Llama 3.3 70B160 GB FP16 · 46 GB 4-bit
- Llama 3.1 405B907 GB FP16 · 253 GB 4-bit
- Llama 4 Scout243 GB FP16 · 68 GB 4-bit
- Llama 4 Maverick894 GB FP16 · 247 GB 4-bit
- Qwen3 8B20 GB FP16 · 6.3 GB 4-bit
- Qwen3 14B34 GB FP16 · 11 GB 4-bit
- Qwen3 32B75 GB FP16 · 22 GB 4-bit
- Qwen3-30B-A3B69 GB FP16 · 20 GB 4-bit
- Qwen3-235B-A22B524 GB FP16 · 145 GB 4-bit
- Qwen2.5 32B75 GB FP16 · 22 GB 4-bit
- Qwen2.5 72B165 GB FP16 · 47 GB 4-bit
- DeepSeek-V3 / R11,492 GB FP16 · 411 GB 4-bit
- gpt-oss-20b47 GB FP16 · 13 GB 4-bit
- gpt-oss-120b260 GB FP16 · 72 GB 4-bit
- Gemma 3 27B62 GB FP16 · 18 GB 4-bit
- Mistral Small 3.2 24B55 GB FP16 · 16 GB 4-bit
- Mixtral 8x7B105 GB FP16 · 30 GB 4-bit
- Phi-4 14B35 GB FP16 · 11 GB 4-bit
- GLM-5.31,654 GB FP16 · 455 GB 4-bit
- LFM2.5-24B-A2B54 GB FP16 · 15 GB 4-bit
- Kolibri-1174 GB FP16 · 48 GB 4-bit
LLM VRAM questions
How much VRAM do I need to run an LLM?
Add up the weights, the KV cache and some headroom. Weights take about 2 GB per billion parameters at FP16/BF16, 1 GB at FP8 and 0.55 GB at 4-bit. The KV cache grows with context length and with the number of concurrent users. Then allow about 10% for activations and fragmentation (vLLM uses 90% of GPU memory by default). Llama 3.3 70B, for example, needs about 160 GB at FP16 for one user at an 8K context.
How do I calculate KV cache size?
KV cache per token = 2 (keys and values) × layers × KV heads × head dimension × bytes per value. Multiply by the context length and by the number of concurrent sequences. For Llama 3.3 70B at FP16 that is 2 × 80 layers × 8 KV heads × 128 head dim × 2 bytes = 327,680 bytes (0.33 MB) per token, so an 8K-token conversation holds 2.7 GB and 100 users at 4K hold 134 GB. Models with multi-head latent attention (DeepSeek-V3) or sliding-window layers (Gemma 3, gpt-oss, Llama 4) cache much less; the calculator handles both.
How many GPUs do I need for a 70B model?
Llama 3.3 70B at FP16/BF16 needs about 160 GB for one user at an 8K context: 2 GPUs (tensor parallel 2) on H100 80 GB. For 100 concurrent users at a 4K context it needs 306 GB (141.2 GB of weights plus 134.2 GB of KV cache, plus headroom): 4 GPUs (tensor parallel 4) on H100, or 2 GPUs (tensor parallel 2) on B200. FP8 weights with an FP8 KV cache bring that to 153 GB: 2 GPUs (tensor parallel 2) on H100.
Which LLMs fit on a 24 GB RTX 4090 or a 32 GB RTX 5090?
At 4-bit with one user at an 8K context, a single RTX 4090 fits Llama 3.1 8B, Qwen3 8B, Qwen3 14B, Qwen3 32B, Qwen3-30B-A3B, Qwen2.5 32B, gpt-oss-20b, Gemma 3 27B, Mistral Small 3.2 24B, Phi-4 14B and LFM2.5-24B-A2B. A single RTX 5090 also fits Mixtral 8x7B. Larger models need several cards with a layer split (llama.cpp, for example) or a Mac Studio with more unified memory.
How much of a Mac Studio's memory can an LLM use?
macOS lets the GPU use about 75% of unified memory by default, keeping the rest for the system. That is about 72 GB of a 96 GB, 192 GB of a 256 GB and 384 GB of a 512 GB Mac Studio. The calculator counts that share, not the full memory.
Do mixture-of-experts models need less VRAM?
No: a mixture-of-experts model needs memory for every expert, because any token can be routed to any of them. DeepSeek-V3 activates 37B of its 671B parameters per token, but all 671B must be resident. Active parameters set speed; total parameters set memory.
Why does tensor parallelism use 1, 2, 4 or 8 GPUs?
Tensor parallelism splits each attention layer's heads evenly across GPUs, so the GPU count must divide both the attention-head and KV-head counts, and serving stacks use powers of two within an 8-GPU server. Phi-4 14B has 10 KV heads, for example, so it can run at tensor parallel 1 or 2, not 4. Workstation cards and Macs usually split by layers instead, which works with any number of cards.
Can I size a model that isn't in the list?
Yes. Type its Hugging Face ID (org/model) or paste its link, and the calculator reads the parameter count and config.json from the Hub. For a gated model, paste its config.json, or enter layers, attention heads, KV heads and head dimension by hand.
Need to deploy private AI?
We design it, validate it, source it, deploy it and help run it.
Give us your workload, users, budget and timeline. Within 48 hours we'll give you a validated cloud/on-prem architecture, BOM, expected costs and sourcing options.