How Many GPUs Do I Need to Run a 70B Parameter Model

Running a large open-weight model yourself, rather than calling a hosted API, shifts the relevant constraint from per-token pricing to raw GPU memory — and figuring out how many GPUs a given model actually needs starts with a genuinely simple weight-size calculation before any of the more complex inference details come into play.

The base VRAM formula

A model’s raw weight size in memory is approximately: Parameters × Bytes per Parameter. At 16-bit precision (FP16 or BF16, the common default for inference), each parameter takes 2 bytes, so a 70-billion-parameter model needs roughly 70 × 2 = 140GB just to hold the weights in memory — before accounting for anything else the model needs during actual inference.

Why the real number is higher than the raw weight size

Loading the weights alone isn’t sufficient to actually run inference — a working model also needs memory for the KV cache (which stores intermediate attention computations for the tokens processed so far in a given request, growing as the conversation or context gets longer) and general activation memory during the forward pass. This overhead commonly adds another 15-25% on top of the raw weight size, meaning a more realistic VRAM estimate for the 70B model is roughly 140GB × 1.2 = 168GB, using a 20% overhead assumption.

Translating VRAM needs into a GPU count

Once you have a total VRAM estimate, dividing by a single GPU’s available memory tells you the minimum number of GPUs needed, rounded up — a fractional GPU obviously isn’t purchasable. At 168GB total, an 80GB professional GPU (like an A100 or H100) needs 168 ÷ 80 = 2.1, rounding up to 3 GPUs minimum. The same 168GB requirement on 24GB consumer-grade GPUs needs 168 ÷ 24 = 7 GPUs — a dramatically higher count, illustrating why serious large-model inference work overwhelmingly uses professional data-center GPUs with much larger per-card memory rather than consumer cards, despite consumer cards’ lower individual cost.

It’s worth being clear that this “divide and round up” calculation gives a genuine minimum, not necessarily a comfortable working number — real multi-GPU setups typically want some additional headroom beyond the bare minimum for practical reasons (batch processing multiple requests simultaneously, longer context windows than the baseline estimate assumed, and general operational margin), so a real deployment often provisions somewhat more than the theoretical minimum GPU count.

Quantization changes the math dramatically

Quantization — running a model at reduced numeric precision, most commonly 8-bit or 4-bit instead of the 16-bit default — directly and proportionally reduces the bytes-per-parameter figure in the base formula, and therefore the total VRAM requirement. Moving the same 70B model from 16-bit (2 bytes/parameter) to 8-bit (1 byte/parameter) roughly halves the raw weight memory requirement, from 140GB down to about 70GB — potentially reducing the GPU count needed from 3 down to 1 on an 80GB card, once the same overhead percentage is reapplied to the smaller base figure. Moving further to 4-bit quantization (0.5 bytes/parameter) roughly quarters the original 16-bit requirement.

This is exactly why quantization is such a heavily used technique for running large models on more modest hardware — it’s a direct, proportional lever on the single biggest cost driver (raw parameter memory), though it does come with some tradeoff in output quality, generally more noticeable at more aggressive quantization levels (4-bit and below) than at 8-bit, which many practitioners find has only a modest, often acceptable quality impact for a substantial memory savings.

Sizing your own setup

The LLM VRAM Calculator on this site applies this exact parameters-times-bytes-times-overhead formula directly — enter a model’s parameter count, your target precision, and an overhead assumption to get a concrete VRAM estimate before deciding on GPU hardware. If you’re training or fine-tuning rather than just running inference, memory and cost requirements scale differently — the GPU Training Cost Calculator covers that separate calculation, and the Inference Latency Calculator addresses the related but distinct question of how fast a given hardware setup can actually generate tokens once it’s running. The Understanding AI Token Costs guide on this site covers the API-based alternative to self-hosting, useful for comparing the two approaches’ real costs directly.