LLaVA

LLaVA 1.0 (April 2023) pioneered the open-weight vision-language model architecture that became the industry standard. The linear projector maps visual patch embeddings from the vision encoder directly into the LLM's token embedding space — the simplest possible vision-language connection. This two-stage recipe established the template that [Qwen-VL](/families/qwen-vl) and [InternVL 1.5+](/families/internvl) later adopted. LLaVA 1.5 (October 2023) improved the architecture with three key changes: (1) MLP projector replacing the linear projection (two-layer MLP with GELU activation for better cross-modal alignment); (2) CLIP ViT-L/14 vision encoder with 336×336 input resolution (up from 224×224); (3) training on the LLaVA-1.5 dataset with academic-task-oriented VQA data (OKVQA, A-OKVQA, GQA, TextCaps). Language model upgraded to Vicuna-1.5 (7B/13B). The MLP projector became the standard VLM connection design, adopted by nearly every subsequent open-weight VLM. This was the first open-weight VLM with multi-scale visual processing, predating [InternVL 2.0](/families/internvl) by 6 months. The language model could be swapped between Mistral-7B, [Vicuna-13B](/families/vicuna), and Nous-Hermes-2-Yi-34B — establishing LLaVA as a language-model-agnostic VLM architecture. The Yi-34B variant was the most capable, demonstrating LLaVA's architecture could scale to 34B language models. LLaVA-OneVision (August 2024) unified single-image, multi-image, and video understanding with interleaved text-image-video inputs. A Qwen-2-7B/72B language model backbone was adopted, moving LLaVA from Vicuna-base to [Qwen-base](/families/qwen) LLMs. The vision encoder remained CLIP ViT-L/14. Video frames were processed as image grids with temporal attention. LLaVA-NeXT-Video (December 2024) extended the OneVision architecture with dedicated video temporal modeling. The architectural evolution from LLaVA 1.0 to LLaVA-OneVision traces the maturation of the VLM field: from simple linear projection + frozen components to MLP projectors + dynamic resolution + model-scale flexibility + multimodal unification. LLaVA defined the open-weight VLM architecture that InternVL, Qwen-VL, and most other VLMs adopted. The core insight — connect a vision encoder to an LLM through a learned projector and jointly fine-tune — remains the standard VLM recipe in 2026.

LLaVA models are moderately locally-friendly, with the 7B-13B variants running on consumer hardware and larger variants requiring multi-GPU setups. LLaVA 1.5 7B at FP16 (~14 GB LLM + 850 MB vision encoder) fits on [RTX 4090](/hardware/rtx-4090) 24 GB with ~9 GB headroom. At Q4, the LLM (~4.5 GB) + vision encoder (850 MB FP16 recommended) fits on [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb) with 10 GB headroom. On [MacBook Pro M4 Max](/hardware/macbook-pro-16-m4-max), ~30 tokens/second via [MLX-LM](/tools/mlx-lm). The 7B is deployable on laptop-class hardware for visual chat. LLaVA 1.5 13B at FP16 (~26 GB LLM + 850 MB vision encoder) requires [RTX 4090 48GB](/hardware/rtx-4090) or 2× [RTX 4090](/hardware/rtx-4090). At Q4 (~8 GB LLM + vision), fits on [RTX 4090](/hardware/rtx-4090) 24 GB with ~14 GB headroom. On [Apple M3 Ultra](/hardware/apple-m3-ultra), ~20 tokens/second. LLaVA-NeXT with Yi-34B at FP16 (~68 GB LLM + 850 MB vision encoder) requires [A100 80GB](/hardware/nvidia-a100-80gb-sxm). At Q4 (~19 GB LLM + vision), fits on [RTX 4090](/hardware/rtx-4090) 24 GB. Dynamic resolution (4 grids) adds ~3 GB intermediate memory, still fitting in 24 GB. On [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra), ~12 tokens/second with dynamic resolution. LLaVA-OneVision with Qwen-2-72B at FP16 (~144 GB LLM + vision) requires 2× [H100 SXM](/hardware/nvidia-h100-sxm). At Q4 (~40 GB LLM), requires 2× [RTX 4090](/hardware/rtx-4090) or [A6000](/hardware/nvidia-a40) 48 GB. The 72B variant is not practically deployable on single consumer GPU. The trajectory: LLaVA 7B/13B are universally deployable; LLaVA-NeXT with Yi-34B is the practical ceiling for single-GPU local VLM deployment; LLaVA-OneVision 72B is multi-GPU. Compared to [InternVL](/families/internvl), LLaVA uses a much smaller vision encoder (428M vs 6B), making vision encoding faster (0.3s vs 3s) at the cost of visual detail quality. This makes LLaVA more suitable for real-time visual chat and less suitable for high-resolution document OCR. LLaVA's small vision encoder is the differentiating factor for local deployment — the faster encoding latency makes visual conversation feel more interactive. For hardware recommendations: [RTX 4090](/hardware/rtx-4090) 24 GB for LLaVA 1.5 13B at Q4 with dynamic resolution; [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb) for LLaVA 1.5 7B at Q4; [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) for LLaVA-NeXT Yi-34B at Q4 with full dynamic resolution.

LLaVA 1.0 (April 2023) scored 90.9% on ScienceQA IMG (image-based science questions) — the first open-weight VLM to exceed 90% on this benchmark. The 158K instruction dataset's quality was validated by these results. LLaVA 1.5 7B (October 2023) scored MME 1,510 points (Perception), MMBench 64.3%, and LLaVA-Bench (In-the-Wild) 67.5% — competitive with early proprietary VLMs. The 13B variant scored MME 1,531, MMBench 67.7%. TextVQA 58.2% (scene text understanding). DocVQA was not benchmarked at this scale — document understanding was not a LLaVA 1.5 priority. LLaVA-NeXT 34B (January 2024) scored MME 1,780, MMBench 81.5%, and SEED-Bench 72.6%. The dynamic resolution improved OCR benchmarks: TextVQA 64.2%, DocVQA 74.3%, ChartQA 69.1%. The Yi-34B language model backbone contributed to reasoning-heavy visual benchmarks: MMMU 46.2%, MathVista 44.8%. These scores were competitive with [InternVL 1.5](/families/internvl), which launched simultaneously with larger vision encoder but similar LLM scale. LLaVA-OneVision 72B (August 2024) scored MME 2,102, MMBench 85.2%, MMMU 55.1%, MathVista 58.3%, DocVQA 82.1%, ChartQA 78.3%. The Qwen-2-72B language model backbone provided superior reasoning and multilingual capability compared to the Yi-34B backbone. Multi-image benchmarks: MMIU (multi-image understanding) 68.2%, video understanding (VideoMME) 58.3%. The LLaVA benchmark identity: pioneering but not benchmark-leading. LLaVA established the VLM benchmarks and methodology that other families use, but [InternVL 2.5](/families/internvl) now leads on most visual benchmarks (particularly OCR/documents) due to the larger vision encoder. Where LLaVA wins: visual conversation naturalness (the original VLM fine-tuned for conversation quality), multi-image reasoning, and video understanding. Where LLaVA trails: OCR and document understanding (behind [InternVL](/families/internvl) by 10-15 points on DocVQA/ChartQA) and visual knowledge (behind [Gemma 3 27B](/families/gemma) on MMMU). LLaVA's benchmark position reflects its role as the community-driven VLM rather than the infrastructure-backed leader.

LLaVA runtime support is the broadest of any VLM family due to its architectural simplicity and community adoption. The simple vision-encoder + MLP-projector + LLM architecture means every engine supports LLaVA with minimal adaptation. [vLLM](/tools/vllm) added LLaVA support in v0.5.0 with the LLaVA multimodal pipeline. LLaVA-NeXT 34B on [L40S](/hardware/nvidia-l40s) with vLLM achieves ~3,000 tokens/second at batch 32. Dynamic resolution grid processing is handled by vLLM's image processor. vLLM supports all LLaVA variants (1.0 through OneVision). LLaVA 1.5 7B on [RTX 4090](/hardware/rtx-4090) via Ollama achieves ~45 tokens/second. LLaVA-NeXT and OneVision require the llava-llama or llava-mistral model types in Ollama. Ollama is the recommended local deployment path for LLaVA — it handles vision preprocessing, chat template, and multimodal inference in one command. LLaVA 1.5 7B at Q4 on [Apple M3 Ultra](/hardware/apple-m3-ultra) with Metal achieves ~20 tokens/second. The llava backend handles the CLIP vision encoder using llama.cpp's CLIP implementation, including 336×336 resolution and MLP projector. [MLX-LM](/tools/mlx-lm) provides good LLaVA support on Apple Silicon. LLaVA 1.5 13B at Q4 on [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) achieves ~18 tokens/second. MLX-LM's LLaVA support includes the CLIP encoder in MLX format. [Transformers](/tools/transformers) (HuggingFace) provides first-class LLaVA support via the `llava`, `llava-next`, and `llava-onevision` model tags. The HuggingFace LLaVA implementation is the most feature-complete, supporting all LLaVA variants, dynamic resolution, and multi-image inputs. [TensorRT-LLM](/tools/tensorrt-llm) supports LLaVA for production NVIDIA deployment. LLaVA-NeXT 34B FP8 on [H100 SXM](/hardware/nvidia-h100-sxm) achieves ~8,000 tokens/second at batch 64. [Text Generation Inference](/tools/text-generation-inference) supports LLaVA via TGI 2.0+ multimodal pipeline. TGI handles LLaVA's image preprocessing and dynamic resolution. [SGLang](/tools/sglang) supports LLaVA 1.5 and NeXT with visual token handling. SGLang's structured generation for LLaVA supports JSON output for VQA tasks. The runtime support breadth reflects LLaVA's position as the reference VLM architecture — any VLM inference support in any engine starts with LLaVA support, and other VLMs ([InternVL](/families/internvl), [Qwen-VL](/families/qwen-vl)) benefit from engines supporting the LLaVA architectural pattern.

LLaVA quantization follows standard VLM patterns with specific considerations for the small vision encoder. GGUF via [llama.cpp](/tools/llama-cpp) is the standard local path. LLaVA 1.5 7B at Q4_K_M (~4.5 GB LLM + 850 MB vision encoder at FP16) = ~5.4 GB total. Quality loss: ~0.4% on MMBench — effectively lossless. The vision encoder is typically kept at FP16/FP32 due to its small size — the 850 MB savings from vision encoder quantization are negligible relative to the visual quality risk. LLaVA-NeXT 34B at Q4_K_M (~19 GB LLM + 850 MB vision encoder) = ~20 GB total. Quality loss: ~0.5% on MMBench. At Q3_K_M, degradation is ~2.2% — acceptable for casual visual chat but not document analysis. The CLIP ViT-L/14 vision encoder (428M) is small enough that quantization benefits are marginal. At Q4, the encoder shrinks from 850 MB to ~250 MB — a 600 MB savings. At Q3, ~200 MB. Given that visual feature quality is central to VLM performance and the savings are small, most LLaVA deployments keep the vision encoder at FP16. The LLM component follows standard dense transformer quantization. GPTQ and AWQ for LLaVA LLM components work normally. AWQ 4-bit for the LLM is lossless (<0.2% degradation on text benchmarks). Vision encoder AWQ is under-tested because the encoder's small size makes precision optimization unnecessary. FP8 via [TensorRT-LLM](/tools/tensorrt-llm) and [vLLM](/tools/vllm) is the datacenter format. LLaVA-NeXT 34B FP8 LLM is effectively lossless. Vision encoder at FP8 is also lossless for the small CLIP ViT-L/14. KV-cache quantization is important for LLaVA because visual tokens consume context. At dynamic resolution (4 grids × 576 tokens/grid + 1 global view × 576 tokens = 2,880 visual tokens), nearly 3K context positions are consumed by the image. 4-bit KV-cache quantization saves ~75% KV-cache memory for these visual tokens. Practical rule: LLaVA 1.5 7B at Q4_K_M is the local sweet spot. LLaVA-NeXT 13B-34B at Q4_K_M is the production floor. Vision encoder at FP16 is recommended for all variants — the memory savings from quantizing a 428M encoder are too small to justify visual quality risk.

LLaVA's ecosystem is the most established in open-weight VLM, benefiting from being the first open-weight VLM architecture and the community's adoption of LLaVA's design patterns. Derivative model volume is moderately high (hundreds of fine-tunes). The LLaVA fine-tuning recipe (PEFT on projector + selective LLM fine-tuning) is the most widely adopted VLM fine-tuning approach. License clarity: Apache 2.0 for most LLaVA variants. The original LLaVA model weights and code are Apache 2.0, permitting commercial use, redistribution, and modification. This is a permissive license tier. The base language models (Vicuna, Yi, Qwen-2) carry their own licenses — LLaVA-34B using Yi-34B inherits Yi's Apache 2.0 license, and LLaVA-OneVision 72B using Qwen-2-72B inherits [Qwen](/families/qwen)'s license. Community adoption is LLaVA's ecosystem superpower. LLaVA established the open-weight VLM architecture standard (vision encoder → projector → LLM) that every subsequent VLM followed. The community's familiarity with LLaVA's architecture means engines, tools, and fine-tuning recipes all support LLaVA first. New VLM capabilities (dynamic resolution, multi-image, video) often debut in LLaVA before propagating to other VLM families. Training data transparency: the LLaVA dataset (158K visual instructions) is fully released under CC-BY-4.0. LLaVA-1.5's dataset (665K) is released. This dataset transparency is a major ecosystem contribution — the LLaVA datasets have been used to train hundreds of VLMs beyond the official LLaVA models. The dataset release is more transparent than [InternVL](/families/internvl) (dataset not released) and most other VLM families. The open-source community culture around LLaVA is strong. LLaVA development is led by academic researchers (primarily University of Wisconsin-Madison, Microsoft Research) with open-source ethos. The code, models, datasets, and training recipes are all publicly available. Ecosystem gaps: LLaVA's visual quality is constrained by the CLIP ViT-L/14 vision encoder (428M), which is 10× smaller than [InternVL](/families/internvl)'s InternViT-6B. LLaVA cannot match InternVL on OCR and document understanding without a vision encoder upgrade. Production deployment tooling (LMDeploy for InternVL, NIM for Nemotron) is less mature for LLaVA — the ecosystem prioritizes research over production.

LLaVA fine-tuning is the most documented and accessible VLM fine-tuning path, with the LLaVA team's two-stage recipe serving as the community standard. LoRA and QLoRA via HuggingFace PEFT is the standard approach. For LLaVA-NeXT 13B, QLoRA on [RTX 4090](/hardware/rtx-4090) uses ~22 GB VRAM and trains in ~8 hours. The recommended LoRA targets: MLP projector (critical — this is the vision-language bridge), LLM Q/K/V/O attention projections, and optionally LLM FFN gate/up layers. The vision encoder is always frozen — its small size (428M) and general visual features don't benefit from domain-specific fine-tuning. LLaVA-NeXT 34B QLoRA requires ~28 GB VRAM — [A6000](/hardware/nvidia-a40) 48 GB or [L40S](/hardware/nvidia-l40s) 48 GB. Training 5K samples takes ~12 hours. The 34B is the practical fine-tuning ceiling for single-GPU setups. Full fine-tuning: LLaVA 1.5 7B needs ~56 GB VRAM (1× [A100 80GB](/hardware/nvidia-a100-80gb-sxm)). LLaVA-NeXT 13B needs ~104 GB VRAM (2× [A100 80GB](/hardware/nvidia-a100-80gb-sxm)). LLaVA-NeXT 34B needs ~272 GB VRAM (4× [A100 80GB](/hardware/nvidia-a100-80gb-sxm)). Full VLM fine-tuning is research-lab or enterprise scale. The LLaVA dataset ecosystem is the richest in VLM: LLaVA-Instruct-158K (the original), LLaVA-1.5-665K (expanded), ShareGPT4V (1.2M), ALLaVA (1.3M), and domain-specific VQA datasets. The LLaVA fine-tuning recipe works on any VLM dataset formatted as image + conversation. Multi-image fine-tuning uses the LLaVA-OneVision dataset format. LLaVA fine-tuning tooling: LLaVA's official GitHub repository provides fine-tuning scripts for each LLaVA variant. HuggingFace TRL supports LLaVA fine-tuning with VLM-specific configurations. Axolotl has LLaVA configs for 1.5 and NeXT variants. The official LLaVA scripts remain the recommended fine-tuning path due to correct handling of image preprocessing, dynamic resolution, and chat template formatting. Vision projector-only fine-tuning is the fastest path to domain adaptation: freeze the vision encoder and LLM, train only the MLP projector (~20M parameters). This requires minimal VRAM (8 GB for LLaVA 7B) and trains in ~1 hour for 5K samples. The quality improvement is limited but for domain-specific visual tasks (e.g., recognizing specific document layouts), projector-only fine-tuning can be sufficient.

LLaVA's Apache 2.0 license for the LLaVA code and weights is permissive for commercial use. Verify the specific LLaVA variant's language model license before commercial deployment. LLaVA-OneVision 72B using Qwen-2-72B is Apache 2.0. LLaVA-NeXT 34B using Yi-34B is Apache 2.0. Chat template: LLaVA uses simple USER/ASSISTANT formatting: `USER: <image> {text} ASSISTANT:`. The `<image>` token is replaced with visual features by the model infrastructure. This simple template is easy to integrate but less standardized than ChatML. [Ollama](/tools/ollama) and [vLLM](/tools/vllm) handle LLaVA's chat template automatically. The CLIP ViT-L/14 vision encoder operates at 336×336 maximum resolution (LLaVA 1.5+). Using incorrect image preprocessing produces severe visual quality degradation — the vision encoder outputs meaningless features. Dynamic resolution (LLaVA-NeXT) processes multiple 336×336 grids; the grid layout is determined by aspect ratio. Vision encoder compute cost is low (0.3s per image on [RTX 4090](/hardware/rtx-4090)) due to the small 428M encoder. This makes LLaVA suitable for real-time visual chat applications — the time-to-first-token includes ~0.3s vision encoding + ~0.5s text generation start, under 1 second total. No other VLM with competitive quality achieves this latency. Multi-image handling in LLaVA-OneVision processes images sequentially, concatenating visual tokens before the text tokens. The maximum number of images per conversation varies by engine: vLLM supports ~16 images, Ollama supports ~4 images, llama.cpp supports ~2 images. Multi-image VRAM scales linearly with image count (~2,880 tokens × N images for dynamic resolution). A known deployment issue on vLLM: the dynamic resolution grid layout in LLaVA-NeXT uses a specific aspect-ratio-based algorithm that differs from [InternVL](/families/internvl)'s tiling. Images with extreme aspect ratios (> 3:1) may not be correctly processed, producing visual distortions. Test LLaVA-NeXT on your target image dimensions before deployment. For applications requiring high-resolution document understanding, InternVL is the stronger choice. For applications requiring interactive visual conversation with low latency, LLaVA's small encoder provides a better user experience. The deployment decision between LLaVA and InternVL is fundamentally: fast visual chat vs high-quality document analysis.