Quick Answer: The GPU VRAM needed for LLM inference depends mainly on model size, weight precision, context length and concurrent requests. A 7B model needs about 14 GB just for FP16 weights, while a 70B model needs about 140 GB. Quantisation can reduce weight memory significantly, but production inference also needs VRAM for the KV cache, runtime buffers and active requests. Always plan for more than the model weights alone.
A 70B model has roughly 140 GB of weights at 16-bit precision.
An NVIDIA H200 has 141 GB of GPU memory.
So, can you simply load the model on one H200 and start serving users?
Usually, not comfortably.
The problem is that model weights are only one part of GPU memory usage. LLM inference also needs memory for the KV cache, activations, framework overhead and concurrent requests.
This is why choosing a cloud GPU based only on the model parameter count can lead to out-of-memory errors or poor production capacity.
If you are still deciding between GPU classes based on workload, memory, performance and deployment requirements, start with the GPU Cloud Buyer’s Guide for India.
This guide focuses specifically on estimating GPU VRAM for LLM inference, from smaller 7B models to 70B and 400B+ models.
LLM VRAM Requirements at a Glance
Start with the model weights.
A simple estimate is:
Model memory ≈ Number of parameters × Bytes per parameter
The approximate storage per parameter is:
| Precision | Approx. Bytes per Parameter |
|---|---|
| FP32 | 4 bytes |
| FP16 / BF16 | 2 bytes |
| FP8 / INT8 | 1 byte |
| INT4 / 4-bit | 0.5 byte |
Using that calculation:
| Model Size | FP32 | FP16 / BF16 | 8-bit | 4-bit |
|---|---|---|---|---|
| 7B | ~28 GB | ~14 GB | ~7 GB | ~3.5 GB |
| 8B | ~32 GB | ~16 GB | ~8 GB | ~4 GB |
| 13B | ~52 GB | ~26 GB | ~13 GB | ~6.5 GB |
| 14B | ~56 GB | ~28 GB | ~14 GB | ~7 GB |
| 32B | ~128 GB | ~64 GB | ~32 GB | ~16 GB |
| 70B | ~280 GB | ~140 GB | ~70 GB | ~35 GB |
| 405B | ~1,620 GB | ~810 GB | ~405 GB | ~202.5 GB |
These are weights-only estimates.
Actual GPU memory usage will normally be higher.
Quantised models may also require additional metadata, scales or higher-precision components, so a 4-bit 70B model should not automatically be treated as exactly 35 GB in production.
Why Model Weights Are Not the Full VRAM Requirement
During inference, GPU memory is typically used by several things:
Model weights + KV cache + runtime memory + temporary activations + serving overhead
For a simple local test, model weights may dominate.
For a production API with long prompts and many simultaneous users, the KV cache can become a major part of the memory requirement.
This means asking:
“Will this model fit on the GPU?”
is different from asking:
“Can this GPU serve this model reliably in production?”
The second question is more important.
How Much VRAM Does a 7B or 8B Model Need?
A 7B model needs roughly:
- 14 GB for FP16/BF16 weights
- 7 GB for 8-bit weights
- 3.5 GB for 4-bit weights
An 8B model is slightly larger:
- 16 GB at FP16/BF16
- 8 GB at 8-bit
- 4 GB at 4-bit
For basic 7B–8B inference, a 24 GB GPU can therefore be a practical starting point when context length and concurrency are moderate.
The NVIDIA L4, for example, provides 24 GB of GPU memory and is positioned more toward inference and efficient AI workloads than large-scale model training.
That does not mean every 8B workload will fit comfortably on every 24 GB configuration. A long context window, large batches or several concurrent requests can still increase memory usage.
Practical starting point
7B–8B FP16: 24 GB GPU for modest inference workloads
7B–8B quantised: Can run with substantially less memory, depending on the framework, context and serving setup
For lightweight APIs, RAG applications, development environments and smaller internal assistants, this model range is often a good place to start before moving to larger GPUs.
How Much VRAM Does a 13B or 14B Model Need?
A 13B model needs approximately:
- 26 GB at FP16/BF16
- 13 GB at 8-bit
- 6.5 GB at 4-bit
A 14B model needs approximately:
- 28 GB at FP16/BF16
- 14 GB at 8-bit
- 7 GB at 4-bit
A 24 GB GPU is therefore usually too small for straightforward FP16 inference because the weights alone can exceed the available memory.
A 48 GB GPU provides much more practical headroom.
The NVIDIA L40S, for example, provides 48 GB of GPU memory and can be useful for inference, generative AI and other workloads that need more memory than 24 GB-class GPUs.
With quantisation, 13B–14B models become much easier to fit into smaller GPU configurations.
Practical starting point
13B–14B FP16: 48 GB-class GPU
13B–14B 8-bit: 24 GB may be workable
13B–14B 4-bit: Smaller-memory GPUs may be sufficient
But capacity is only one consideration. Memory bandwidth and inference performance also matter once request volume increases.
How Much VRAM Does a 30B–32B Model Need?
A 32B model needs approximately:
- 128 GB at FP32
- 64 GB at FP16/BF16
- 32 GB at 8-bit
- 16 GB at 4-bit
This is where GPU choice starts changing significantly.
An 80 GB NVIDIA A100 or NVIDIA H100 can hold approximately 64 GB of 16-bit model weights while leaving some capacity for inference overhead.
However, an 80 GB GPU with 64 GB of weights has much less spare memory than the raw GPU capacity suggests.
Long contexts and production concurrency may still make a larger-memory GPU or quantised model preferable.
Practical starting point
32B FP16: 80 GB-class GPU, with workload testing
32B 8-bit: 48 GB-class GPU can become practical
32B 4-bit: 24 GB may fit the weights, but production headroom still needs testing
If the workload is expected to scale, choose based on the target serving configuration rather than whether a single test request runs successfully.
How Much VRAM Does a 70B Model Need?
This is where memory planning becomes much more important.
A 70B model needs approximately:
- 280 GB at FP32
- 140 GB at FP16/BF16
- 70 GB at 8-bit
- 35 GB at 4-bit
At FP16, the model weights alone are around 140 GB.
The NVIDIA H200 provides 141 GB of HBM3e memory.
At first glance, that looks like a perfect match.
It is not.
If approximately 140 GB is already occupied by model weights, almost no memory remains for the KV cache, runtime buffers and actual inference workload.
For full or near-full 16-bit inference, multi-GPU deployment is normally a more practical approach for a 70B model.
Quantisation changes the picture.
At 8-bit, approximately 70 GB of weight memory can theoretically fit on an 80 GB GPU. But only around 10 GB remains before considering implementation overhead and KV cache requirements.
At 4-bit, approximately 35 GB of weights can fit within a 48 GB GPU.
That can make single-GPU 70B inference possible for some workloads, but “fits into memory” does not automatically mean “good production performance.”
Practical starting point
70B FP16/BF16: Multi-GPU is usually the practical choice
70B 8-bit: 80 GB may fit the weights, but memory headroom can be tight
70B 4-bit: 48 GB can become possible for some inference workloads
For production, benchmark latency, tokens per second, context length and concurrency before deciding.
If you are specifically choosing between Hopper GPUs for a memory-heavy workload, the H100 vs H200 comparison provides a more direct look at their differences in memory capacity, bandwidth and workload fit.
What About 100B, 400B and Larger Models?
The same calculation applies, but multi-GPU infrastructure quickly becomes unavoidable.
Take a 405B dense model.
Approximate weight memory is:
| Precision | Approx. Weight Memory |
|---|---|
| FP16 / BF16 | 810 GB |
| 8-bit | 405 GB |
| 4-bit | 202.5 GB |
Even 4-bit weights exceed the memory capacity of a single NVIDIA B200.
At this scale, infrastructure planning moves beyond selecting one GPU.
You need to evaluate:
- Number of GPUs
- Tensor parallelism
- Pipeline parallelism
- GPU-to-GPU interconnect
- Memory bandwidth
- Network topology
- Inference framework
- KV cache distribution
- Batch scheduling
The cheapest GPU per hour is rarely enough information to choose infrastructure for a model at this scale.
For workloads that include model training or fine-tuning as well as inference, the GPU Cloud for LLM Training in India guide covers the additional memory, networking and multi-GPU requirements involved on the training side.
How Quantisation Reduces LLM VRAM
Quantisation stores model weights using fewer bits.
For example, moving from 16-bit to 8-bit roughly halves raw model-weight memory.
Moving from 16-bit to 4-bit can reduce raw weight storage much further.
For a 70B model:
FP16: ~140 GB
8-bit: ~70 GB
4-bit: ~35 GB
That is why quantisation can change a workload from requiring several GPUs to potentially fitting on one GPU.
But quantisation is not free.
Depending on the model, quantisation method and serving framework, it can affect:
- Accuracy
- Output quality
- Latency
- Throughput
- Kernel support
- Hardware compatibility
Do not choose quantisation only because the model fits.
Benchmark the quantised model against your actual prompts and evaluation dataset.
Why Context Length Increases VRAM Usage
Model weights remain relatively fixed after the model is loaded.
The KV cache does not.
During autoregressive generation, the model stores attention key and value states for previously processed tokens so it does not need to recompute them for every new token.
For many transformer architectures:
More tokens → larger KV cache → more GPU memory
This means a model running comfortably with a 4K-token context may require substantially more memory when deployed with 32K, 64K or longer contexts.
The exact increase depends on the model architecture.
Models using techniques such as grouped-query attention, multi-query attention, sliding-window attention or chunked attention may behave differently.
This is why the model’s advertised maximum context length should not automatically become your production configuration.
If most real requests use 2,000 tokens, provisioning every request for the maximum possible context may waste memory.
Why Concurrency Changes GPU Memory Requirements
Suppose one user sends a request.
The GPU needs memory for that request’s active inference state.
Now suppose 20 users send requests simultaneously.
The model weights are still loaded once, but request-specific memory—especially KV cache—must support many active sequences.
This is why VRAM affects more than whether the model starts.
It also affects how many users the GPU can serve at once.
A GPU with more memory may allow:
- More concurrent requests
- Larger batches
- Longer context windows
- Larger KV cache pools
- Fewer out-of-memory failures
For production LLM inference, these can be more important than simply loading the largest possible model.
Development VRAM vs Production VRAM
A common mistake is sizing production infrastructure using a local test.
For example:
“The model runs on my GPU, so 24 GB is enough.”
That proves the model can run under the test conditions.
It does not prove that 24 GB is enough for production.
| Workload | What Matters Most |
|---|---|
| Local testing | Model fits and generates successfully |
| Internal prototype | Moderate context and a few users |
| Small API | Concurrency, latency and stable memory usage |
| Production SaaS | Throughput, batching, KV cache and peak demand |
| Large inference service | Multi-GPU scaling, scheduling and cost per token |
Always size production against peak realistic usage, not a single prompt.
Which NVIDIA GPU Fits Which VRAM Range?
Here are common data-centre GPU memory capacities used for AI inference.
| GPU | GPU Memory | Typical VRAM Positioning |
|---|---|---|
| NVIDIA L4 | 24 GB | Smaller and quantised models |
| NVIDIA L40S | 48 GB | Mid-sized models and quantised larger models |
| NVIDIA A100 | 80 GB | Larger AI workloads |
| NVIDIA H100 | 80 GB | High-performance AI inference and training |
| NVIDIA H200 | 141 GB | Memory-heavy LLM workloads |
| NVIDIA B200 | 180 GB+ class | Large Blackwell-scale AI workloads |
VRAM capacity alone should not be used to rank these GPUs.
H100, H200 and B200 also differ significantly in architecture, memory bandwidth, tensor performance and supported precision formats.
The correct question is not:
“Which GPU has the most VRAM?”
It is:
“Which GPU gives my workload enough memory and acceptable performance at the lowest total serving cost?”
Once you know the memory range you need, use the getInfra.cloud GPU pricing index to compare GPU availability and public cloud pricing across tracked providers.
What to Do If Your LLM Does Not Fit in VRAM
You have several options.
Quantise the model. Moving from FP16 to 8-bit or 4-bit can dramatically reduce weight memory.
Reduce the context length. Long contexts increase KV cache usage.
Reduce concurrency or batch size. Fewer active sequences reduce request-specific memory pressure.
Use multiple GPUs. Tensor or pipeline parallelism can distribute model weights across GPUs.
Use KV cache optimisation. Some frameworks support paged, quantised or offloaded KV cache approaches.
Offload to CPU memory. This can reduce GPU memory requirements, but moving data between CPU and GPU may reduce performance.
Choose a smaller model. A well-performing 8B or 14B model can sometimes be more economical than operating a much larger model.
For production systems, reducing the model size is often worth considering before simply adding more GPUs.
Don’t Choose a GPU by VRAM Alone
Two GPUs with enough memory can still deliver very different inference performance.
Also check:
Memory bandwidth: LLM inference frequently moves large amounts of model data through GPU memory.
Compute performance: Precision and supported tensor operations affect inference speed.
GPU interconnect: Multi-GPU inference can depend heavily on NVLink, NVSwitch or network topology.
CPU and system RAM: Tokenisation, preprocessing and serving layers still need CPU resources.
Storage: Models need to load quickly, and repeated container or instance starts can make slow storage painful.
Serving framework: vLLM, TensorRT-LLM and other serving stacks manage batching and memory differently.
Cloud pricing: A GPU with a higher hourly rate may still cost less per request if it completes substantially more work.
The final metric should move from ₹ per GPU-hour toward metrics such as:
- Cost per million tokens
- Requests per second
- Tokens per second
- Time to first token
- Maximum stable concurrency
That gives a much better picture of real inference economics.
A Simple VRAM Sizing Process
Before selecting a cloud GPU, document:
1. Model parameter count
Is it 8B, 14B, 32B, 70B or larger?
2. Weight precision
Will you serve FP16/BF16, FP8/INT8 or 4-bit?
3. Context length
What do real requests need—not just the model’s maximum supported context?
4. Concurrency
How many simultaneous active requests do you expect?
5. Output length
Longer generations keep more state active.
6. Serving framework
Memory allocation differs between frameworks.
7. Safety headroom
Do not select a GPU whose entire memory capacity is consumed by model weights on paper.
8. Benchmark
Run realistic prompts, context lengths and concurrency before reserving long-term GPU capacity.
Example: Choosing VRAM for a 70B Model
Suppose you want to serve a 70B model.
Option 1: FP16
Weights alone: ~140 GB
A 141 GB H200 is too close to the weight requirement to assume comfortable production inference.
A multi-GPU configuration is safer.
Option 2: 8-bit
Weights: ~70 GB
An 80 GB GPU can theoretically contain the weights.
But production headroom for KV cache and runtime memory may be limited.
Option 3: 4-bit
Weights: ~35 GB
A 48 GB GPU may become feasible.
That leaves more room for runtime memory, although context length and concurrency still need testing.
The choice therefore depends on more than model size.
It depends on the quality level, latency, throughput and number of users you need to support.
Common VRAM Sizing Mistakes
Avoid these mistakes:
- Matching model weight size exactly to GPU memory
- Ignoring the KV cache
- Testing only one short prompt
- Ignoring concurrent users
- Using the maximum model context unnecessarily
- Assuming all quantisation methods behave the same
- Choosing the largest GPU before benchmarking smaller options
- Looking only at GPU hourly price
- Assuming a model that loads successfully is production-ready
The goal is not to make the model barely fit.
The goal is to make the workload run reliably and economically.
FAQs
How much VRAM does a 7B LLM need?
A 7B model requires roughly 14 GB just for FP16/BF16 weights, around 7 GB at 8-bit and about 3.5 GB at 4-bit. Actual inference needs additional memory for KV cache and runtime overhead, so a 24 GB GPU can be a practical starting point for 7B FP16 inference with moderate context and concurrency.
How much VRAM does a 13B LLM need?
A 13B model requires approximately 26 GB for FP16/BF16 weights, 13 GB at 8-bit and 6.5 GB at 4-bit. A 48 GB GPU gives more practical headroom for 16-bit inference, while quantised versions may fit on GPUs with less memory.
Can a 70B model run on one H100 80 GB?
A 70B model requires approximately 140 GB for 16-bit weights, so it will not fit on one 80 GB H100 at FP16/BF16. At 8-bit, the weight requirement falls to roughly 70 GB, but that leaves limited memory for KV cache and runtime overhead. A 4-bit version requires roughly 35 GB of weight memory and is easier to serve on a single GPU.
Can a 70B FP16 model run on one H200?
A 70B model has approximately 140 GB of weights at FP16, while H200 provides 141 GB of GPU memory. That leaves almost no room for KV cache or serving overhead. In practice, use quantisation or multiple GPUs instead of assuming that weights fitting on paper means the workload will run reliably.
Does longer context length need more VRAM?
Yes. During LLM inference, the KV cache stores information for previously processed tokens. Longer prompts and generations therefore generally increase KV cache memory usage. The exact amount depends on model architecture and cache implementation.
Does more VRAM make LLM inference faster?
Not automatically. More VRAM lets you run larger models, longer contexts or more concurrent requests, but inference speed also depends on memory bandwidth, tensor performance, precision, serving software and batching. A GPU with more memory is not necessarily faster for every workload.
Final Takeaway
Start LLM inference sizing with model weights, but do not stop there.
A rough calculation may tell you that a 70B 8-bit model needs about 70 GB or that a 13B FP16 model needs about 26 GB. Production infrastructure also needs room for KV cache, framework overhead, context length and concurrent requests.
For most teams, the safer process is:
Estimate → add workload requirements → shortlist GPUs → benchmark → measure cost per useful output.
Once you know the approximate VRAM range your model needs, compare cloud GPU pricing and availability by memory, performance, region and provider rather than choosing on GPU name alone.
