Diffusion Model

A diffusion model is a type of generative model that learns to reverse a gradual noising process. During training, the model learns to predict and remove noise added to data (e.g., images) at various timesteps. At inference, it starts from pure random noise and iteratively denoises it step by step to produce a coherent output. For local AI operators, diffusion models are the architecture behind image generators like Stable Diffusion and FLUX. They require significant VRAM (e.g., 8–24 GB for standard models) and benefit from GPU acceleration; generation speed depends on the number of denoising steps (typically 20–50) and model size.

Diffusion models work by defining a forward process that adds Gaussian noise to data over T timesteps, turning it into pure noise. The model is trained to reverse this process: given a noisy sample at timestep t, it predicts the noise that was added. At inference, the model starts with random noise and iteratively applies the learned denoising step, often using a scheduler (e.g., DDPM, DDIM, or Euler) that controls the step size and noise schedule. Variants like latent diffusion (used in Stable Diffusion) perform the process in a compressed latent space to reduce computational cost. Key parameters for operators include the number of inference steps (more steps = higher quality but slower), guidance scale (how strongly the model follows a prompt), and the sampler choice. Diffusion models are distinct from autoregressive models (which generate tokens sequentially) and GANs (which use adversarial training). They are now the dominant architecture for text-to-image and text-to-video generation.

Running Stable Diffusion 3.5 Medium (2.7B parameters) on an RTX 4090 (24 GB VRAM) takes about 2–4 seconds per image at 30 denoising steps. On an RTX 3060 (12 GB VRAM), the same model may need 6–10 seconds and might require offloading some layers to system RAM if VRAM is tight. Using a smaller model like SDXL Turbo (1.3B) with fewer steps (4–8) can cut generation time to under a second on the 4090.

In LM Studio or ComfyUI, an operator loads a diffusion model checkpoint (e.g., a .safetensors file for Stable Diffusion). They set the number of inference steps (e.g., 20), the sampler (e.g., Euler Ancestral), and the guidance scale (e.g., 7.5). The UI shows VRAM usage and estimated time per generation. After clicking 'Generate', the model iteratively denoises a random latent tensor, and the final image is displayed. Operators often experiment with different samplers and step counts to balance quality and speed.