Latent Space

Latent space is the internal, compressed representation of data that a generative model learns during training. It is a lower-dimensional space where each point (a latent vector) encodes the essential features of a data point (e.g., an image, text, or audio). Operators encounter latent space when using models like Stable Diffusion or LLMs: the model takes an input, maps it to a latent vector, then decodes that vector to generate output. The structure of latent space determines how the model interpolates, combines concepts, and controls generation (e.g., via prompt engineering or latent manipulation).

In generative AI, latent space is the abstract, high-dimensional space where the model stores its learned representations. For example, in a variational autoencoder (VAE) used in Stable Diffusion, an input image is encoded into a latent vector (e.g., 4x64x64 for SD1.5), which is then decoded to reconstruct the image. The latent space is continuous and structured so that similar data points are close together. Operators manipulate latent space through techniques like prompt engineering (which guides the latent vector via cross-attention), latent blending (interpolating between two latent vectors), or using LoRAs (which modify the latent space locally). The size of the latent space (e.g., the number of latent dimensions) affects VRAM usage and generation speed—larger latent spaces require more memory and compute.

When generating an image with Stable Diffusion in Automatic1111, the model first encodes your prompt into a latent vector (e.g., 64x64x4 for SDXL). The diffusion process then denoises this latent, step by step, in latent space—not in pixel space. The final latent is decoded into a 1024x1024 image. The VRAM required depends on the latent size: SDXL's larger latent (compared to SD1.5) means ~2 GB more VRAM usage during generation.

In ComfyUI, you can directly manipulate latent space by adding noise, blending latents from different prompts, or using a 'Latent Interpolation' node. For example, to create a morph between two concepts, you generate two latents (e.g., 'cat' and 'dog'), then linearly interpolate between them in latent space before decoding. In llama.cpp, the model's internal hidden states (the last hidden layer) are a form of latent space—operators can extract them via --embedding to get a vector representation of a text.