How to install NVIDIA Container Toolkit for Docker

What this does

Installs the NVIDIA Container Toolkit, enabling Docker containers to access NVIDIA GPU hardware for accelerated compute tasks. After completion, Docker containers can use --gpus flag to schedule workloads on the GPU.

Steps

1. Add the NVIDIA Container Toolkit repository.

   distribution=$(. /etc/os-release && echo $ID$VERSION_ID)
   curl -fsSL https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-toolkit.gpg
   echo "deb [signed-by=/usr/share/keyrings/nvidia-toolkit.gpg] https://nvidia.github.io/nvidia-container-runtime/${distribution} /" | sudo tee /etc/apt/sources.list.d/nvidia-toolkit.list
   

   Expected output: Repository entry created.

2. Install the nvidia-container-toolkit package.

   sudo apt-get update
   sudo apt-get install -y nvidia-container-toolkit
   

   Expected output: Package installed without errors.

3. Configure Docker to use the NVIDIA runtime.

   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker
   

   Expected output: Docker configuration updated and service restarted.

• Record the local run evidence. Save the exact command, runtime or package version, model name if applicable, and observed output so the result can be reproduced later.

Verification

docker run --rm nvidia/cuda:12.4-base nvidia-smi
# Expected: GPU table printed (device name, memory, driver version, CUDA version)

Common failures

• could not select runtime driver — Docker restart was skipped. Run sudo systemctl restart docker again.
• nvidia-smi: command not found inside container — The runtime configuration step was omitted. Re-run sudo nvidia-ctk runtime configure --runtime=docker.
• Repository GPG key error — Create the keyring directory with sudo mkdir -p /usr/share/keyrings.
• Package not found — Check /etc/os-release and substitute values manually if needed.
• permission denied when running nvidia-ctk — The command requires root. Prefix with sudo and retry.

Operator checkpoint

Before treating this as solved, write down the local runtime, model or package version, hardware/backend if relevant, and the verification output. This keeps the guide useful as a Will-It-Run style decision instead of a one-off command transcript.

Related guides

• How to verify CUDA setup for AI workloads
• How to configure PyTorch with GPU support
• Course Local AI Fundamentals