Multimodal AI

Multimodal AI refers to models that process and generate multiple data types—typically text, images, and sometimes audio or video—within a single architecture. Unlike text-only LLMs, multimodal models can accept image inputs (e.g., a photo) and answer questions about them, or generate images from text descriptions. For operators, this means running models like LLaVA, Qwen-VL, or GPT-4V locally requires handling both text and vision encoders, which increases VRAM usage and inference latency. A typical multimodal pipeline encodes the image into embeddings via a vision encoder (e.g., CLIP), then feeds those embeddings into the language model alongside text tokens.

Multimodal models combine separate encoders for each modality (vision, text, audio) with a shared language model backbone. The most common architecture for image+text models is a vision encoder (like CLIP ViT) that converts images into a sequence of embeddings, which are then projected into the language model's embedding space via a connector (e.g., a linear layer or Q-Former). During inference, the model processes both text tokens and image embeddings together, allowing it to answer visual questions or describe scenes. Variants include LLaVA (simple connector), Qwen-VL (cross-attention), and GPT-4V (proprietary). For operators, the key practical difference is that multimodal models require additional VRAM for the vision encoder (typically 1-4 GB) and have longer context windows due to image tokens (e.g., 256 tokens per image). Running them on consumer GPUs often requires quantization of both the vision encoder and the language model to fit within VRAM limits.

A 16 GB RTX 4060 can run LLaVA 1.6 7B at Q4_K_M (~5 GB for the LLM + ~1 GB for the CLIP vision encoder) with a 4K context, achieving ~20 tok/s. The same card cannot run LLaVA 13B Q4 because the LLM alone uses ~8 GB, leaving insufficient VRAM for the encoder and context. On an Apple M2 Max with 32 GB unified memory, LLaVA 13B Q4 runs at ~15 tok/s via MLX, using about 12 GB total.

In LM Studio, loading a multimodal model like LLaVA shows two model files: the vision encoder (e.g., clip-vit-large-patch14) and the language model (e.g., llava-v1.6-mistral-7b). When you drag an image into the chat, the runtime encodes it into embeddings before generating a response. In llama.cpp, you run ./llama-cli -m llava-v1.6-mistral-7b.Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image photo.jpg -p "Describe this image" to invoke the multimodal pipeline.