How to mount GPU in Docker container

What this does

Enables a Docker container to access the host GPU so AI workloads run on dedicated hardware rather than the CPU. After this guide the container will render nvidia-smi output or AMD equivalent inside the container.

Steps

1. Install the container toolkit on the host. GPU passthrough requires the host runtime to inject GPU access into containers.

   distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
   curl -fsSL https://nvidia.github.io/nvidia-docker/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
   curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
   sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker
   

   Expected output: nvidia-container-toolkit writes updated daemon JSON config.

2. Launch the container with the GPU flag. The --gpus flag routes GPU access into the container environment.

   docker run --gpus all -it --rm nvidia/cuda:12.6.3-base-ubuntu24.04 nvidia-smi
   

   Expected output: Table showing GPU model, driver version, memory usage.

3. Verify AMD GPU passthrough on supported hosts. ROCm uses different runtime flags.

   docker run --device=/dev/kfd --device=/dev/dri -it --rm rocm/rockml:latest rocm-smi
   

   Expected output: ROCm SMI table displaying GPU card and VRAM.

• Record the local run evidence. Save the exact command, runtime or package version, model name if applicable, and observed output so the result can be reproduced later.

Verification

docker run --gpus all --rm ubuntu:24.04 nvidia-smi
# Expected: full GPU table with driver version, CUDA version, and GPU index

Common failures

• Unknown runtime specified "nvidia" — nvidia-container-toolkit was not installed or Docker daemon was not restarted. Re-run installation steps and sudo systemctl restart docker.
• NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver — GPU drivers are not loaded on the host. Run nvidia-smi outside the container first.
• Permission denied for /dev/dri — User is not in the docker group. Run sudo usermod -aG docker $USER and log back in.
• GPU not visible inside container despite no error — GPU compute mode may be off. Run sudo nvidia-smi -pm ENABLED on the host before launching.
• Container runs but GPU reports 0% utilization — The workload may be CPU-bound or the model is too small. Check nvidia-smi pmon for process activity.

Related guides

• How to build custom Docker image for AI models
• How to persist Ollama data in Docker
• Course Local AI Fundamentals