DirectML

What it is and how it works

DirectML is Microsoft's hardware-accelerated machine learning API built on top of DirectX 12. Rather than being a standalone inference engine with its own model format and runtime, it's a low-level compute library that sits below higher-level frameworks — most commonly exposed to end users through the DirectML execution provider (EP) in ONNX Runtime, and also usable directly via DirectX 12 compute shaders for developers who want to hand-roll GPU kernels. The core idea is that DirectML translates ML operators (convolutions, matrix multiplies, attention blocks, normalization layers) into DirectX 12 compute workloads, which the GPU vendor's DirectX driver then schedules on the hardware. Because every GPU sold for Windows already ships a conformant DirectX 12 driver, DirectML gets vendor-agnostic acceleration "for free" — no separate CUDA toolkit, no ROCm stack, no proprietary NPU SDK install required for basic operation.

This architecture is fundamentally different from CUDA or ROCm, which are vendor-owned stacks with kernels hand-tuned by NVIDIA and AMD engineers respectively for their own silicon. DirectML instead relies on the DirectX driver layer as the abstraction boundary: Microsoft defines the operator set and dispatch semantics, and NVIDIA, AMD, Intel, and Qualcomm each implement DirectX 12 compute support in their own drivers. The practical effect is that the same DirectML/ONNX Runtime code path runs across NVIDIA GeForce/RTX cards, AMD Radeon GPUs, Intel Arc/integrated graphics, and — critically for the current Windows-on-ARM push — Qualcomm Snapdragon X Elite NPUs and Intel Lunar Lake NPUs. This last point is DirectML's real differentiator in 2026: it's one of the few inference paths with genuine first-class NPU support across multiple silicon vendors, since Microsoft has been actively co-engineering DML drivers with Qualcomm and Intel for the Copilot+ PC NPU story.

The tradeoff for this vendor-agnostic reach is that DirectML operators are generic — they're not hand-tuned per-architecture the way a CUDA kernel authored by NVIDIA for a specific SM version is, or the way a ROCm kernel is tuned for a specific CDNA/RDNA generation. DirectML depends on the GPU vendor's driver-level compiler and scheduler to extract performance from a fairly generic operator graph, and historically that has left measurable throughput on the table compared to native vendor stacks.

Deployment patterns

The overwhelmingly dominant deployment pattern for DirectML is the Windows desktop or laptop, single-user, non-CUDA GPU scenario: someone with an AMD Radeon card, an Intel Arc GPU, or a Copilot+ PC with a Qualcomm/Intel NPU who wants local inference without installing ROCm (which has historically had spotty Windows support) or hunting for Intel-specific runtimes. In practice this means installing onnxruntime-directml via pip, pointing an ONNX Runtime InferenceSession at the DML execution provider, and running an ONNX-exported model — commonly a quantized Stable Diffusion pipeline, a smaller transformer model exported through Olive (Microsoft's model optimization toolchain) or Optimum, or a vision/ASR model. Microsoft's own tooling (Olive, the ONNX Runtime GenAI package) targets DirectML as one of its EP backends specifically to make this export-and-run flow relatively turnkey for Windows developers.

A secondary pattern is the Windows-on-ARM / Copilot+ PC path, where DirectML is effectively the only practical way for a third-party app to reach the NPU without writing to Qualcomm's or Intel's native NPU SDK directly. App developers shipping Windows Store or sideloaded AI features (local image generation, background blur, transcription) route through DirectML/ONNX Runtime to get NPU offload across hardware they don't control.

What you essentially never see is DirectML in a homelab multi-GPU server or team-shared inference box — those deployments are Linux-first and go straight to CUDA or ROCm, both of which have more mature multi-GPU orchestration, better batching support, and tighter integration with the serving stacks (vLLM, TGI, llama.cpp's CUDA/HIP backends) that this niche actually uses. DirectML's serving story is essentially "single machine, single user, ONNX Runtime session" — there is no DirectML-native equivalent of a production LLM server with continuous batching.

How it compares

Against CUDA, DirectML is the clear underdog on raw throughput and ecosystem depth — CUDA has over a decade of vendor-tuned kernels, is the default target for essentially every major inference and training framework, and is the only realistic choice if you own an NVIDIA card and care about maximum performance. DirectML's value proposition only kicks in when CUDA isn't an option, i.e., non-NVIDIA hardware.

Against ROCm, the comparison is closer to "different tradeoffs, same audience." ROCm is AMD's native stack and, when it works, generally edges out DirectML on AMD hardware because it's tuned closer to the metal. But ROCm's Windows support has historically lagged its Linux support significantly, and its supported-GPU list has been narrower. DirectML wins on Windows specifically because it just works across AMD's whole GPU line without checking a compatibility matrix.

Against llama.cpp (which has its own Vulkan and, on Windows, CPU/CUDA/SYCL backends), the comparison is about maturity of LLM-specific optimization. llama.cpp's backends have deep, LLM-specific kernel tuning (fused attention, optimized quantized matmuls, KV-cache handling) built by a community laser-focused on transformer inference. DirectML, by contrast, is a general ML operator library first — LLM-specific fusions and quantization tricks arrive later and via other layers (ONNX Runtime GenAI, Olive) rather than being native to DML itself, which is exactly the "LLM-specific optimizations behind vLLM / llama.cpp" gap noted in its cons.

Best use cases and honest limitations

DirectML is the right call for Windows-only developers and users who need one code path to reach whatever GPU or NPU happens to be in the machine — AMD, Intel, Qualcomm, or NVIDIA — without maintaining separate vendor-specific build configs, and especially for anyone targeting Copilot+ PC NPU acceleration, where it's close to the only accessible route today. It's also a reasonable default for ONNX-based vision, speech, and diffusion workloads shipped as part of a Windows application, since the "pre-installed on Windows 10+" DirectX 12 dependency means zero extra runtime install burden for end users.

It's the wrong call for anyone on Linux or macOS (no path exists at all), anyone chasing maximum single-GPU throughput on NVIDIA hardware (use CUDA directly), anyone running serious LLM inference who wants the latest quantization and batching tricks (llama.cpp or vLLM will be ahead), and anyone building multi-GPU or server-grade deployments, where DirectML's single-session desktop-oriented design doesn't fit. The realistic 15-30% throughput gap versus native CUDA/ROCm paths, combined with the Windows-only ceiling, means DirectML should be understood as a compatibility and reach play rather than a performance play — it exists to make "it runs on whatever GPU is in this Windows PC" true, not to be the fastest option available.