RUNLOCALAIv38
->Will it run?Best GPUCompareTroubleshootStartLearnPulseModelsHardwareToolsBench
Run check
RUNLOCALAI

Independently operated catalog for local-AI hardware and software. Hand-written verdicts. Source-cited claims. Reproducible commands when we have them.

OP·Eruo Fredoline
DIR
  • Models
  • Hardware
  • Tools
  • Benchmarks
TOOLS
  • Will it run?
  • Compare hardware
  • Cost vs cloud
  • Choose my GPU
  • Prompting kits
  • Quick answers
REF
  • All buyer guides
  • Learn local AI
  • Methodology
  • Glossary
  • Errors KB
  • Trust
EDITOR
  • About
  • Author
  • How we make money
  • Editorial policy
  • Contact
LEGAL
  • Privacy
  • Terms
  • Sitemap
MAIL · MONTHLY DIGEST
Get monthly local AI changes
Monthly recap. No spam.
DISCLOSURE

Some links on this site are affiliate links (Amazon Associates and other first-class retailers). When you buy through them, we earn a small commission at no extra cost to you. Affiliate links do not influence our verdicts — there are cards we rate highly that we don't have affiliate relationships with, and cards that sell well that we refuse to recommend. Read more →

© 2026 runlocalai.coIndependently operated
RUNLOCALAI · v38
  1. >
  2. Home
  3. /Learn
  4. /Courses
  5. /Local AI on Linux
  6. /Ch. 1
Local AI on Linux

01. Why Linux for AI

Chapter 1 of 15 · 10 min
KEY INSIGHT

Linux is the only OS where you control every layer from the kernel ABI to the container runtime, giving you full GPU access with zero virtualization overhead.

Linux dominates AI infrastructure because every major ML framework treats it as the primary target. CUDA, ROCm, cuDNN, and TensorRT are developed and tested on Linux first. The Linux kernel exposes GPU devices through /dev/nvidia* and /dev/dri/* in ways that container runtimes can bind-mount or pass through with zero configuration overhead.

The GPU vendor driver stacks integrate directly with the kernel ABI. On Ubuntu, nvidia-driver-535 installs a kernel module that lives at /usr/lib/modules/$(uname -r)/kernel/drivers/video/nvidia.ko. On Fedora, the same package uses DKMS to rebuild the module against the running kernel after every kernel update. If you run uname -r and it returns 6.8.0-45-generic, the driver must match that exact kernel ABI. Mismatches are the leading cause of GPU initialization failures.

Linux gives you nvidia-smi from the command line, /proc/driver/nvidia/ for diagnostics, and nvidia-ctk on newer stacks. On no other OS can you cat /proc/driver/nvidia/gpus/0000:01:00.0/information to retrieve GPU serial numbers programmatically. This matters for multi-GPU racks where you need to identify which card is which from SSH.

Containers are first-class citizens on Linux. docker run --gpus all works because the NVIDIA container runtime hooks into the Docker daemon via --hooks-dir=/usr/share/containers/oci/hooks.d/. On Windows or macOS, this requires a Linux VM under the hood. The performance penalty is real and measurable.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.

EXERCISE

Run ls /dev/nvidia* and nvidia-smi on a Linux machine with a GPU. Note the driver version, CUDA version, and device index. Document what you see.

← Overview
Local AI on Linux
Chapter 2 →
NVIDIA Driver Installation