Skip to content

Tools / LLM GPU calculator / RTX 4090

What LLMs can run on an RTX 4090?

Which of 22 open LLMs fit one RTX 4090, and how many it takes for the rest, at FP16, FP8 and 4-bit.

Size your own scenario

Short answer

One RTX 4090 (24 GB) runs 11 of the 22 open LLMs sized here at 4-bit, 5 at FP8 and 2 at FP16, for one user at an 8K context. At 4-bit: Llama 3.1 8B, Qwen3 8B, Qwen3 14B, Phi-4 14B, gpt-oss-20b, LFM2.5-24B-A2B, Mistral Small 3.2 24B, Gemma 3 27B, Qwen3-30B-A3B, Qwen3 32B and Qwen2.5 32B. The largest that fits one card is Qwen3 32B (32.8B parameters), at 4-bit. Bigger models split their layers across several cards: Mixtral 8x7B takes 2 at 4-bit.

Updated . One user, 8K context, KV cache at FP16, plus 10% headroom for activations and the runtime.

Every model on the RTX 4090

ModelFP16/BF16FP84-bit (INT4)
Llama 3.1 8B1 card19 GB1 card10 GB1 card6.1 GB
Qwen3 8B1 card20 GB1 card10 GB1 card6.3 GB
Qwen3 14B2 cards34 GB1 card18 GB1 card11 GB
Phi-4 14B2 cards35 GB1 card18 GB1 card11 GB
gpt-oss-20b2 cards47 GB1 card23 GB1 card13 GB
LFM2.5-24B-A2B3 cards54 GB2 cards27 GB1 card15 GB
Mistral Small 3.2 24B3 cards55 GB2 cards28 GB1 card16 GB
Gemma 3 27B3 cards62 GB2 cards32 GB1 card18 GB
Qwen3-30B-A3B3 cards69 GB2 cards35 GB1 card20 GB
Qwen3 32B4 cards75 GB2 cards39 GB1 card22 GB
Qwen2.5 32B4 cards75 GB2 cards39 GB1 card22 GB
Mixtral 8x7B5 cards105 GB3 cards53 GB2 cards30 GB
Llama 3.3 70B7 cards160 GB4 cards81 GB2 cards46 GB
Qwen2.5 72B7 cards165 GB4 cards84 GB2 cards47 GB
Kolibri-18 cards174 GB4 cards87 GB2 cards48 GB
Llama 4 Scout11 cards243 GB6 cards122 GB3 cards68 GB
gpt-oss-120b11 cards260 GB6 cards130 GB3 cards72 GB
Qwen3-235B-A22B22 cards524 GB11 cards263 GB7 cards145 GB
Llama 4 Maverick38 cards894 GB19 cards448 GB11 cards247 GB
Llama 3.1 405B38 cards907 GB19 cards456 GB11 cards253 GB
DeepSeek-V3 / R163 cards1,492 GB32 cards746 GB18 cards411 GB
GLM-5.369 cards1,654 GB35 cards827 GB19 cards455 GB

Counts are for a layer split, which works with any number of cards. Below each count is the memory needed: (weights + KV cache) ÷ 0.9. Capacity only: speed depends on memory bandwidth and the serving stack.

Memory is step one. Throughput, latency, power and the network decide the real build.

Size my actual workload

RTX 4090 specifications

Memory
24 GB GDDR6X
Memory bandwidth
1,008 GB/s
Typical system
Desktop PC
Counted as
Layer split across cards (llama.cpp style)

Questions about the RTX 4090

What LLMs can run on an RTX 4090?

For one user at an 8K context, these fit one RTX 4090. At 4-bit: Llama 3.1 8B, Qwen3 8B, Qwen3 14B, Phi-4 14B, gpt-oss-20b, LFM2.5-24B-A2B, Mistral Small 3.2 24B, Gemma 3 27B, Qwen3-30B-A3B, Qwen3 32B and Qwen2.5 32B. At FP8: Llama 3.1 8B, Qwen3 8B, Qwen3 14B, Phi-4 14B and gpt-oss-20b. At FP16/BF16: Llama 3.1 8B and Qwen3 8B.

What is the largest LLM an RTX 4090 can run?

Of the models sized here, for one user at an 8K context: at 4-bit, Qwen3 32B (32.8B parameters, 22 GB); at FP8, gpt-oss-20b (20.9B parameters, 23 GB); at FP16, Qwen3 8B (8.19B parameters, 20 GB). Mixture-of-experts models count every expert, because all of them must be in memory.

How many RTX 4090 cards does Llama 3.3 70B need?

For one user at an 8K context it needs 160 GB at FP16/BF16, 81 GB at FP8 and 46 GB at 4-bit: 7 cards at FP16, 4 cards at FP8 and 2 cards at 4-bit, with a layer split.

Can an RTX 4090 run DeepSeek-V3 / R1?

Not on one card: with all 671B parameters resident it needs 746 GB at FP8 and 411 GB at 4-bit for one user at an 8K context. That is 32 cards at FP8 and 18 cards at 4-bit, with a layer split across more cards than one workstation holds.

Can I combine several RTX 4090 cards?

Yes, with a layer split: each card holds some of the model's layers, as llama.cpp does. Memory adds up across them; speed does not, because they take turns on each token. The counts on this page assume a layer split.

Other GPUs

Different context length, more users, or a model that isn’t listed? The LLM GPU calculator sizes any Hugging Face model for any of these GPUs.

Need to deploy private AI?

We design it, validate it, source it, deploy it and help run it.

Get my 48-hour blueprint

Give us your workload, users, budget and timeline. Within 48 hours we'll give you a validated cloud/on-prem architecture, BOM, expected costs and sourcing options.