HOW-TO · SET

How to install NVIDIA Container Toolkit for Docker

intermediate15 minBy Eruo Fredoline
Target environment
Ubuntu 24.04 · Ollama 0.4.xWindows 11 · Ollama 0.4.xmacOS 15 · Ollama 0.4.x
PREREQUISITES

Docker installed and running, NVIDIA GPU with compatible drivers

What this does

Installs the NVIDIA Container Toolkit, enabling Docker containers to access NVIDIA GPU hardware for accelerated compute tasks. After completion, Docker containers can use --gpus flag to schedule workloads on the GPU.

Steps

  1. Add the NVIDIA Container Toolkit repository.

    distribution=$(. /etc/os-release && echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-toolkit.gpg
echo "deb [signed-by=/usr/share/keyrings/nvidia-toolkit.gpg] https://nvidia.github.io/nvidia-container-runtime/${distribution} /" | sudo tee /etc/apt/sources.list.d/nvidia-toolkit.list

    Expected output: Repository entry created.

  2. Install the nvidia-container-toolkit package.

    sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

    Expected output: Package installed without errors.

  3. Configure Docker to use the NVIDIA runtime.

    sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

    Expected output: Docker configuration updated and service restarted.

  • Record the local run evidence. Save the exact command, runtime or package version, model name if applicable, and observed output so the result can be reproduced later.

Verification

docker run --rm nvidia/cuda:12.4-base nvidia-smi
# Expected: GPU table printed (device name, memory, driver version, CUDA version)

Common failures

  • could not select runtime driver — Docker restart was skipped. Run sudo systemctl restart docker again.
  • nvidia-smi: command not found inside container — The runtime configuration step was omitted. Re-run sudo nvidia-ctk runtime configure --runtime=docker.
  • Repository GPG key error — Create the keyring directory with sudo mkdir -p /usr/share/keyrings.
  • Package not found — Check /etc/os-release and substitute values manually if needed.
  • permission denied when running nvidia-ctk — The command requires root. Prefix with sudo and retry.

Operator checkpoint

Before treating this as solved, write down the local runtime, model or package version, hardware/backend if relevant, and the verification output. This keeps the guide useful as a Will-It-Run style decision instead of a one-off command transcript.

Related guides

RELATED GUIDES
SET
How to configure PyTorch with GPU support
SET
How to verify CUDA setup for AI workloads
← All how-to guidesCourses →