Estimate the quantized weights, add context-cache and runtime memory, then compare with usable accelerator memory. A 5 GB model file does not imply a 5 GB GPU can run the complete workload. These are inference planning calculations for one sequence, not measured benchmarks or training requirements.
1. Estimate weight memory
For a first approximation, multiply parameters by effective bits per weight and divide by eight. Quantization formats carry scales and other data, so “4-bit” does not mean every parameter occupies exactly four bits in the finished file.
Weight memory in decimal GB ≈ parameter count in billions × effective bits / 8MyLocalRig uses planning factors of 4.83 bits for Q4_K_M, 5.67 for Q5_K_M, and 8.5 for Q8_0, plus an allowance for additional tensors. Actual file size depends on the architecture and quantizer. Use the chosen file’s size and metadata when available.
| Dense parameter count | Q4_K_M weights | Q5_K_M weights | Q8_0 weights |
|---|---|---|---|
| 8B | ~4.8 GB | ~5.7 GB | ~8.5 GB |
| 14B | ~8.5 GB | ~9.9 GB | ~14.9 GB |
| 32B | ~19.3 GB | ~22.7 GB | ~34.0 GB |
| 70B | ~42.3 GB | ~49.6 GB | ~74.4 GB |
This table excludes context cache, runtime allocations, and display use. GB here means one billion bytes; a runtime may display GiB instead. Avoid comparing those units as though they were identical.
2. Add the context cache
During autoregressive text generation, attention keys and values consume memory. A conventional full-attention estimate for a single sequence using a two-byte cache is:
KV bytes ≈ 2 × layers × KV heads × head dimension × context tokens × 2For an illustrative model with 32 layers, 8 KV heads, head dimension 128, and 8,192 context tokens, this gives 1,073,741,824 bytes: about 1.07 GB or 1 GiB. At 32,768 tokens, the same simple cache estimate is about 4.29 GB. The model weights have not changed.
Sliding-window attention, hybrid architectures, cache quantization, batching, and runtime allocation strategies change the calculation. Vision models can also need an image encoder and image-token memory. When architecture fields are missing, MyLocalRig flags the omitted cache rather than claiming a complete estimate.
3. Build a complete planning budget
Using the illustrative 8B model at Q4_K_M and 8K context: weights are about 4.83 GB, an additional tensor allowance is 0.1 GB, cache is about 1.07 GB, and the checker adds 0.7 GB of working memory. That totals roughly 6.7 GB before considering the device reserve. These allowances are assumptions, not universal runtime requirements.
The checker also reserves part of nominal device memory. Another app or display workload can consume more than the allowance. A near-limit result should lead to a smaller context test and inspection of actual memory use. Do not turn a rough estimate into a purchase guarantee.
4. Distinguish capacity from speed
CPU offload can make a model usable when all its weights do not fit on the accelerator. System RAM and VRAM remain different memory pools with different speeds. Increasing RAM can prevent a load failure without delivering the generation speed of a full GPU load.
A mixture-of-experts model still needs storage for all its expert weights unless a specific runtime uses a verified alternate strategy. Active parameter count is not the amount of model memory to budget. MyLocalRig counts total parameters for capacity and does not apply its simple dense-model speed formula to MoE models.
5. Validate before upgrading
- Choose the exact model and quantization you intend to use.
- In My rig, select your hardware and read the Q4_K_M / 8K planning assumptions above the results.
- Read the row’s fit reason. If memory is tight, test a shorter context in your runtime or choose a smaller model.
- Load the matching variant in your runtime and inspect the actual context and processor placement.
- Only then decide whether you need more accelerator memory, a smaller model, or simply a shorter context.
The “full-memory quant” column considers the accelerator pool, or system RAM for CPU inference. It does not present a precision level requiring CPU offload as a full-GPU fit. Image, speech, and video pipelines require their own workflow-specific memory budgets.
Sources and review notes
Reviewed October 21, 2026. Official documentation and publisher data support the setup details. Memory examples are calculations, not firsthand benchmark results. Versions, defaults, and hardware support can change.