NVIDIA, AMD and Intel: the local-LLM software stacks

CUDA, ROCm, Vulkan and SYCL — how the vendor paths compare for running LLMs locally, and why the fastest backend is now architecture-dependent (RDNA3 vs RDNA4). See the living status board for dated, sourced detail. Reviewed July 2026.

2 min read Reviewed July 2026

A GPU's raw specs are only half the story for local LLMs; the software stack decides what actually runs and how fast. Here is the practical picture as of July 2026.

NVIDIA — CUDA

The most mature path by a wide margin. CUDA has the broadest support across runtimes and unlocks the accelerations others lack: FlashAttention, TensorRT-LLM, ExLlamaV2, first-class vLLM, and NVLink for pooling memory across cards. If you want the fewest surprises, this is the default.

AMD — ROCm (and Vulkan)

AMD Radeon cards run today's models, but the stack is less mature and platform matters:

  • ROCm on Linux is the most complete AMD path. But the fastest backend depends on your architecture: on RDNA3 prefer ROCm/HIP, while on RDNA4 a Vulkan backend can be faster (measured token generation).
  • ROCm on Windows works too, and here the picture has moved further: on RDNA4 a Vulkan backend is often ahead of ROCm/HIP, not behind it. There is no single fixed Vulkan-vs-ROCm ratio, and it changes over time — see the AMD & Intel viability status board for the dated, sourced picture.

Intel — SYCL and Vulkan

Intel Arc also runs models, with two backends — SYCL / IPEX-LLM and Vulkan — and which is faster varies by metric and build rather than following a fixed ratio: SYCL/IPEX-LLM can dominate prompt processing while Vulkan can lead decode. Resizable BAR is effectively mandatory. See the status board for dated, sourced entries.

Vulkan — the common denominator

Vulkan runs on all three vendors. It is often close to — and sometimes ahead of — the vendor-native stack (notably on RDNA4, and on Windows), while staying portable and, on AMD/Intel, frequently the most reliable option.

What the gap actually costs you

Non-NVIDIA cards have no FlashAttention-3, no TensorRT-LLM, no ExLlamaV2, and no NVLink pooling. Crucially, those mostly affect prompt processing and features, not raw decode: token generation is bandwidth-bound and that roofline holds across vendors (Bandwidth vs compute). So an AMD/Intel card can decode at the speed its bandwidth implies while still trailing on long-prompt ingestion (pp vs tg).

Each AMD and Intel GPU page in the catalog carries a dated ROCm/Vulkan caveat consistent with this, and links to the living AMD & Intel viability status board — updated as the stack moves. This guidance was reviewed July 2026.