On a 8 GB GPU, 12 catalog models run fully on the GPU at an 8,192-token context. The most capable is Qwen3-8B at Q4_K_M (needs ~6.8 GiB). Pick a smaller model or a lower quant for more headroom.
Figures assume a 8 GB GPU paired with 32 GB of system RAM (a typical desktop), judged at an 8,192-token context. Speed depends on the specific card — this page ranks by fit, not speed.
TL;DR
8 GB of VRAM is the mainstream entry point for local LLMs, and it's genuinely usable. Today's small instruct models run fully on the GPU at the standard 4-bit sweet-spot quants, and that's exactly what the live list below ranks — largest first, with the quant that fits. If a model you want isn't listed, a smaller sibling or a lower quant is usually the honest fix.
The weights aren't the whole budget: the KV cache grows with context length, so a model that fits at this page's default context can stop fitting if you push the context much higher. At this tier, quant choice matters more than at any other — stepping down one rung often turns "doesn't fit" into "runs fully on GPU".
Qwen3-8B at Q4_K_M · 8.2B params · needs ~6.8 GiB
One command to run it (llama.cpp):
llama-server -m Qwen_Qwen3-8B-Q4_K_M.gguf -c 8192 -ngl 999| Model | Sweet-spot quant | Fits in |
|---|---|---|
| Qwen3-8B Qwen | Q4_K_M Runs fully on GPU | ~6.8 GiB |
| DeepSeek-R1-Distill-Llama-8B deepseek-ai | Q4_K_M Runs fully on GPU | ~6.6 GiB |
| gemma-4-E4B-it google | QAT-Q4_0 Runs fully on GPU | ~5.9 GiB |
| DeepSeek-R1-Distill-Qwen-7B deepseek-ai | Q4_K_M Runs fully on GPU | ~5.7 GiB |
| Qwen3-Embedding-8B Qwen | Q4_K_M Runs fully on GPU | ~6.4 GiB |
| gemma-4-E2B-it google | Q8_0 Runs fully on GPU | ~5.8 GiB |
| Qwen3-Embedding-4B Qwen | Q4_K_M Runs fully on GPU | ~4.2 GiB |
| Phi-4-mini-instruct microsoft | Q8_0 Runs fully on GPU | ~5.7 GiB |
| SmolLM3-3B HuggingFaceTB | BF16 Runs fully on GPU | ~7.4 GiB |
| dots.ocr rednote-hilab | BF16 Runs fully on GPU | ~7 GiB |
| DeepSeek-R1-Distill-Qwen-1.5B deepseek-ai | Q8_0 Runs fully on GPU | ~2.7 GiB |
| Qwen3-Reranker-0.6B Qwen | BF16 Runs fully on GPU | ~2.6 GiB |
Derived live from the fit engine + catalog at an 8,192-token context. "Fits in" is the modelled VRAM the sweet-spot quant needs (weights + KV cache + overhead). Speed depends on your specific card — check a GPU page or the calculator.
Yes. Current small instruct models run fully on an 8 GB GPU at 4-bit sweet-spot quants and are practical for chat, drafting, and light coding help. You won't run large models entirely on the GPU at this tier, but the models that do fit are far more capable than the small models of a year or two ago.
You can — llama.cpp and most desktop runtimes support CPU+GPU splits — but generation slows down sharply once part of the model leaves the GPU. The table on this page deliberately lists only models that run fully on the GPU; treat offloading as a fallback for occasional use, not a daily driver.
Yes. The KV cache grows with context, so a model that fits at this page's assumed context can run out of memory at a much longer one. If you want long conversations or big documents, check your exact model and context in the fit calculator before downloading.
As a rule of thumb, take the highest quant that still fits with headroom at your target context — the table shows a sweet-spot quant per model, and the quant helper ranks every quant of a model for your card. Below roughly 4-bit, a smaller model at a healthier quant usually beats a bigger model crushed to very low bits.
This page ranks by fit, not speed — two 8 GB cards can generate at very different rates because memory bandwidth differs. Check your card's GPU page or the calculator for speed context on your exact setup.
selected to compare · pick at least 2