HOW-TO · SET

How to mount GPU in Docker container

intermediate10 minBy Eruo Fredoline
Target environment
Ubuntu 24.04 · Ollama 0.4.xWindows 11 · Ollama 0.4.xmacOS 15 · Ollama 0.4.x
PREREQUISITES

Docker installed, NVIDIA Container Toolkit or AMD ROCm

What this does

Enables a Docker container to access the host GPU so AI workloads run on dedicated hardware rather than the CPU. After this guide the container will render nvidia-smi output or AMD equivalent inside the container.

Steps

  1. Install the container toolkit on the host. GPU passthrough requires the host runtime to inject GPU access into containers.

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/nvidia-docker/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

    Expected output: nvidia-container-toolkit writes updated daemon JSON config.

  2. Launch the container with the GPU flag. The --gpus flag routes GPU access into the container environment.

    docker run --gpus all -it --rm nvidia/cuda:12.6.3-base-ubuntu24.04 nvidia-smi

    Expected output: Table showing GPU model, driver version, memory usage.

  3. Verify AMD GPU passthrough on supported hosts. ROCm uses different runtime flags.

    docker run --device=/dev/kfd --device=/dev/dri -it --rm rocm/rockml:latest rocm-smi

    Expected output: ROCm SMI table displaying GPU card and VRAM.

  • Record the local run evidence. Save the exact command, runtime or package version, model name if applicable, and observed output so the result can be reproduced later.

Verification

docker run --gpus all --rm ubuntu:24.04 nvidia-smi
# Expected: full GPU table with driver version, CUDA version, and GPU index

Common failures

  • Unknown runtime specified "nvidia" — nvidia-container-toolkit was not installed or Docker daemon was not restarted. Re-run installation steps and sudo systemctl restart docker.
  • NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver — GPU drivers are not loaded on the host. Run nvidia-smi outside the container first.
  • Permission denied for /dev/dri — User is not in the docker group. Run sudo usermod -aG docker $USER and log back in.
  • GPU not visible inside container despite no error — GPU compute mode may be off. Run sudo nvidia-smi -pm ENABLED on the host before launching.
  • Container runs but GPU reports 0% utilization — The workload may be CPU-bound or the model is too small. Check nvidia-smi pmon for process activity.

Related guides

RELATED GUIDES
SET
How to build custom Docker image for AI models
SET
How to persist Ollama data in Docker
← All how-to guidesCourses →