InternVL

InternVL 1.0 (December 2023) launched as OpenGVLab's first open-weight vision-language model. Architecture: ViT-L/14 vision encoder (428M, from InternViT-6B pretraining) with a Q-Former-style cross-attention bridge to an InternLM-7B language model. The Q-Former bridge compressed 256 visual tokens into 96 query tokens attending to the vision encoder output, then projected them into the LLM's embedding space. InternVL 1.5 (April 2024) simplified the architecture: the Q-Former bridge was replaced with a two-layer MLP projector, aligning InternVL with the LLaVA architecture that was proving dominant. The vision encoder was upgraded to InternViT-6B (6B parameters) — the largest dedicated vision encoder in any open-weight VLM at release. Training: the vision encoder was pre-trained on 6B image-text pairs, then the full model was instruction-tuned on 4M visual instruction examples. The LLM component used InternLM2-7B or InternLM2-20B. InternVL 2.0 (August 2024) expanded the family with 1B, 2B, 4B, 8B, 26B, and 76B variants. Architecture: InternViT-6B vision encoder + MLP projector + InternLM2 language model at various scales. The 76B variant was the largest dense open-weight VLM at release, using InternLM2-76B. Training data scaled to 22M visual instruction examples. Dynamic resolution support was added: images are processed at native aspect ratio with multiple crops (up to 40 tiles of 448×448), enabling high-resolution document and chart understanding. [InternVL 2.5](/families/internvl) (January 2025) represents the current generation. Architecture: InternViT-6B-448px-V2.5 (6B vision encoder with 448×448 native resolution), MLP projector, and InternLM2.5 language models (1B to 78B). The 78B variant uses InternLM2.5-78B as the LLM backbone. InternVL 3.0 (expected late 2026) is anticipated to add video understanding and 3D vision capabilities. The architectural evolution: InternVL started with a Q-Former bridge (complex, unique), simplified to MLP projector (following community standards), and then scaled both the vision and language components aggressively. The family's differentiator is the massive InternViT-6B vision encoder — 10× larger than [LLaVA](/families/[llava](/families/llava))'s CLIP ViT-L/14 (428M), providing superior visual understanding at the cost of increased compute for vision encoding.

InternVL models present a bifurcated local viability profile: sub-8B models are highly locally-friendly; 26B+ models require significant hardware, especially with dynamic resolution enabled. InternVL 2.5 1B at FP16 (~2 GB LLM + 850 MB vision encoder) runs on any device. On [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb), ~100 tokens/second. The 1B is an on-device VLM for simple visual tasks. InternVL 2.5 4B at Q4 (~3 GB LLM + 850 MB vision encoder) fits easily on any GPU. InternVL 2.5 8B at Q4 (~5.5 GB LLM + 1.7 GB vision encoder at Q4) fits on [RTX 4090](/hardware/rtx-4090) with ~16 GB headroom. On [Apple M3 Ultra](/hardware/apple-m3-ultra), ~25 tokens/second. The 8B is the recommended local VLM for most use cases — capable document and chart understanding at consumer hardware. InternVL 2.5 26B at Q4 (~16 GB LLM + 3 GB vision encoder at Q4) fits on [RTX 4090](/hardware/rtx-4090) 24 GB with ~4 GB headroom for single-image inference. Dynamic resolution with multiple tiles dramatically increases VRAM: processing 20 tiles of 448×448 adds ~7 GB of intermediate vision encoder memory. At 40 tiles (maximum dynamic resolution), vision encoding alone can consume 14 GB VRAM. For production document understanding at full resolution, the 26B at Q4 with full dynamic resolution requires 32-40 GB VRAM — [RTX 4090 48GB](/hardware/rtx-4090), [A6000](/hardware/nvidia-a40) 48 GB, or [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) 192 GB. InternVL 2.5 78B at Q4 (~44 GB LLM + 3 GB vision encoder) requires 2× [RTX 4090](/hardware/rtx-4090) or [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) 192 GB. With dynamic resolution at 40 tiles, total VRAM consumption can reach ~60 GB — requiring 3× [RTX 4090](/hardware/rtx-4090) or [A100 80GB](/hardware/nvidia-a100-80gb-sxm). The 78B is datacenter-grade for production document understanding at high resolution. The trajectory: InternVL models scale from highly accessible (1B-8B) to hardware-demanding (78B + dynamic resolution). The vision encoder (InternViT-6B) is a fixed compute cost regardless of LLM size — the vision encoder is 6B parameters whether paired with a 1B or 78B LLM. This means the smallest InternVL models pay a disproportionate vision-encoding compute cost (6B vision encoder + 1B LLM), while large models amortize it. For local users, InternVL 2.5 8B provides the best balance of vision capability and VRAM requirements.

InternVL 1.0 (December 2023) scored MME (Multimodal Evaluation) 2,212 points and MMBench 80.2% — competitive with early VLMs. The Q-Former bridge provided strong visual-text alignment but was architecturally complex. InternVL 1.5 (April 2024) scored MME 2,308 points, MMBench 83.5%, and OCRBench 786 — the family's OCR/document understanding strength emerged clearly. The InternViT-6B vision encoder showed particular strength on text-heavy images: DocVQA 90.1%, ChartQA 83.2%, InfoVQA 72.1% — exceeding [LLaVA 1.5](/families/llava) by 10+ points on document benchmarks. InternVL 2.0 76B (August 2024) set new benchmarks: MME 2,398 points, MMBench 89.2%, OCRBench 867, DocVQA 93.5%, ChartQA 88.4%, InfoVQA 80.2%. On MMMU (multimodal understanding with knowledge), 58.7% — matching GPT-4V on multimodal knowledge. MMBench-CN (Chinese multimodal) 88.1%. The 76B variant established InternVL as the open-weight VLM benchmark leader. InternVL 2.5 78B (January 2025) pushed further: MME 2,432 points, MMBench 91.2%, OCRBench 892, DocVQA 94.8%, ChartQA 90.1%, InfoVQA 82.5%. MMMU 62.3% — exceeding GPT-4V on multimodal knowledge. MathVision (visual math reasoning) 68.2% — a new benchmark strength. RealWorldQA (spatial reasoning) 78.3%. InternVL 2.5 8B scored MMBench 84.2%, DocVQA 88.1%, ChartQA 80.2% — competitive with 2023-era 70B+ VLMs at 8B scale, demonstrating the InternViT-6B encoder's disproportionate contribution to quality at small LLM sizes. The InternVL benchmark identity: visual document understanding and OCR are the family's signature strengths, driven by the InternViT-6B vision encoder and dynamic resolution. Where InternVL wins: document VQA, chart understanding, OCR, and Chinese multimodal benchmarks (matching or exceeding GPT-4V). Where InternVL trails: general visual conversation (MMBench scores are strong but behind [Gemma 3 27B](/families/gemma) on natural image understanding), and inference speed (6B vision encoder adds latency vs smaller encoders in [LLaVA](/families/llava) and [Qwen-VL](/families/qwen-vl)).

InternVL runtime support is strongest on [Transformers](/tools/transformers), [vLLM](/tools/vllm), and [LMDeploy](/tools/lmdeploy) (OpenGVLab's own inference engine). Community engine support is growing but less mature than for text-only families. [LMDeploy](/tools/lmdeploy) is the reference InternVL inference engine, developed by OpenGVLab. LMDeploy provides InternVL-optimized inference with TurboMind backend, dynamic resolution tile processing, and 4-bit KV-cache quantization. For production InternVL deployment, LMDeploy is the recommended engine. [vLLM](/tools/vllm) added InternVL support in v0.6.0. Multimodal support handles image preprocessing and tiled encoding. vLLM's prefix caching does not apply to vision tokens, limiting throughput benefits for visual workloads. [SGLang](/tools/sglang) supports InternVL 2.0+. The RadixAttention cache has limited benefit for visual workloads where image tokens vary per request. [Ollama](/tools/ollama) supports InternVL 2.0+ through the llama.cpp backend. `ollama run internvl2:8b` provides local deployment. Multimodal support in Ollama for InternVL handles single-image inference with standard preprocessing. Dynamic resolution with multiple tiles is limited to 6 tiles in Ollama vs 40 tiles in LMDeploy. [llama.cpp](/tools/llama-cpp) supports InternVL 2.0+ through the llava multimodal backend. InternVL 2.5 8B on [Apple M3 Ultra](/hardware/apple-m3-ultra) with Metal achieves ~15 tokens/second. Image encoding uses the llama.cpp CLIP backend; InternViT uses a CLIP-compatible architecture. [TensorRT-LLM](/tools/tensorrt-llm) supports InternVL 2.0+ for production NVIDIA deployment. InternVL 2.5 26B FP8 on [H100 SXM](/hardware/nvidia-h100-sxm) achieves ~8,000 tokens/second at batch 64. [MLX-LM](/tools/mlx-lm) has experimental InternVL support on Apple Silicon. InternVL 2.5 8B on [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) via MLX achieves ~18 tokens/second with single-image inference. [Transformers](/tools/transformers) provides first-class InternVL support. The model loads with AutoModel + processor, handling image preprocessing, tiled encoding, and chat template formatting. This is 10× slower than [LLaVA](/families/llava)'s 428M vision encoder (0.3s encoding). The encoding latency makes InternVL less suitable for real-time visual chat applications and better suited for document analysis where encoding latency is acceptable.

InternVL quantization follows standard VLM patterns with specific considerations for the large vision encoder. GGUF via [llama.cpp](/tools/llama-cpp) is the primary path for local deployment. InternVL 2.5 8B LLM at Q4_K_M (~5.5 GB) + InternViT vision encoder at Q4_K_M (~1.7 GB) = ~7.2 GB total, fitting easily on any modern GPU. Quality loss: ~0.4% on MMBench — effectively lossless for the combined model. InternVL 2.5 26B at Q4_K_M (~16 GB LLM + ~3 GB vision encoder at Q4 = ~19 GB) fits on [RTX 4090](/hardware/rtx-4090). Quality loss ~0.5% on MMBench. At Q3_K_M (~12.5 GB LLM + 2.3 GB vision encoder = ~15 GB), ~2.5% MMBench degradation, borderline acceptable. The InternViT-6B vision encoder at 6B parameters is large enough to benefit from quantization. At Q4_K_M (1.7 GB), quality loss on visual benchmarks is ~0.3%. At Q3 (1.3 GB), ~1.5% visual quality loss. The vision encoder's quantization sensitivity is slightly higher than the LLM's — visual features are more sensitive to precision loss than text token probabilities. For production document OCR, keep the vision encoder at Q4 or above. AWQ and GPTQ support is limited for InternVL. AWQ for the LLM component works normally; vision encoder AWQ is under-tested. FP8 via [TensorRT-LLM](/tools/tensorrt-llm) and LMDeploy is the datacenter format. InternVL 2.5 26B FP8 (~26 GB LLM + 6 GB vision encoder) is effectively lossless (<0.1% degradation visual benchmarks). KV-cache quantization is important for InternVL because vision tokens add to context length. At 40 tiles × 256 tokens/tile, vision tokens alone consume ~10K context positions. 4-bit KV-cache quantization via LMDeploy saves ~75% KV-cache memory, enabling higher batch sizes for multi-image document processing. Practical rule: InternVL 2.5 8B at Q4_K_M is the sweet spot for local deployment. InternVL 2.5 26B at Q4_K_M is the production quality floor. Vision encoder at Q4 is the recommended minimum; Q3 vision encoding degrades OCR quality measurably.

InternVL's ecosystem is the largest in open-weight vision-language AI, with OpenGVLab's sustained investment and strong Chinese academic community support. Derivative model volume is moderate (hundreds of fine-tunes). Key derivatives: Mini-InternVL (smaller variants optimized for edge deployment), InternVL2-Pro (commercial closed-weight extensions), and Chinese-language document understanding fine-tunes on ModelScope. The vision-language fine-tuning community is smaller than text-only communities ([Llama](/families/llama), [Qwen](/families/qwen)) but InternVL is the dominant VLM in academic vision-language research. License clarity: MIT license covers InternVL 1.5 through 2.5. MIT is the most permissive major license — no restrictions beyond attribution and liability disclaimer. This makes InternVL the most commercially-flexible VLM choice, more permissive than [LLaVA](/families/llava) (Apache 2.0) or [Gemma 3](/families/gemma) (Gemma Terms). MIT licensing enables unrestricted commercial deployment of vision-language capabilities. Training data transparency: OpenGVLab discloses training data composition and methodology but does not release actual training datasets. The InternViT-6B pretraining data is partially described. This is comparable to most open-weight families. The OpenGVLab ecosystem (Shanghai AI Lab) provides InternVL-specific tooling: LMDeploy for inference, XComposer for multimodal application development, and InternVL model cards on HuggingFace. The academic community publication ecosystem around InternVL is substantial — InternVL papers are among the most-cited VLM papers in 2024-2026. ModelScope integration is strong for the Chinese ecosystem. InternVL models are hosted on ModelScope with Chinese-language documentation, deployment guides, and fine-tuning recipes. The Chinese VLM deployment community uses InternVL as the primary open-weight option for document understanding. Ecosystem gaps: Western VLM community favors [LLaVA](/families/llava) for historical and community reasons; InternVL's larger vision encoder (6B) makes it less accessible for resource-constrained researchers than [LLaVA](/families/llava) (428M encoder); and the commercial deployment ecosystem (SaaS platforms, enterprise tools) has stronger [Qwen-VL](/families/qwen-vl) integration than InternVL integration.

InternVL fine-tuning follows standard VLM fine-tuning patterns with specific considerations for the large vision encoder. LoRA and QLoRA via HuggingFace PEFT is the standard path. InternVL 2.5 8B QLoRA rank 32 (LLM LoRA + vision projector LoRA, keeping vision encoder frozen) on [RTX 4090](/hardware/rtx-4090) uses ~22 GB VRAM and trains 5K visual instruction samples in ~6 hours. The vision encoder (InternViT-6B) is typically frozen during fine-tuning to preserve visual feature quality — fine-tuning the vision encoder risks degrading the pretrained visual representations that are InternVL's core advantage. InternVL 2.5 26B QLoRA (LLM LoRA only) requires ~30 GB VRAM — [A6000](/hardware/nvidia-a40) 48 GB or 2× [RTX 4090](/hardware/rtx-4090). Training 5K samples takes ~15 hours. Vision projector fine-tuning: the MLP projector between vision encoder and LLM is the primary fine-tuning target for domain-specific visual tasks. LoRA on the projector (~20M parameters) adds minimal VRAM overhead and adapts visual representations to domain-specific content. This is the recommended approach for document-type specialization (legal documents, medical images, engineering diagrams). The dataset ecosystem for VLM fine-tuning includes LLaVA-Instruct (158K visual conversations), ShareGPT4V (1.2M), ALLaVA (1.3M), and domain-specific VQA datasets (DocVQA, ChartQA, InfoVQA training splits). InternVL's dynamic resolution requires tile-aware data preparation — training images should be pre-processed at the target resolution and tile configuration used during inference. LMDeploy provides fine-tuning integration with InternVL: LMDeploy's TurboMind engine can serve fine-tuned InternVL models with optimized throughput. The LMDeploy fine-tuning pipeline handles dynamic resolution training correctly. Axolotl has limited InternVL support; community fine-tuning tooling is VLM-general rather than InternVL-specific. HuggingFace PEFT + transformers is the recommended fine-tuning stack. Full fine-tuning at 26B requires ~200 GB VRAM (3× [A100 80GB](/hardware/nvidia-a100-80gb-sxm)) — research-lab scale. The 8B at full fine-tune requires ~64 GB VRAM (1× [A100 80GB](/hardware/nvidia-a100-80gb-sxm)). For most use cases, QLoRA on the 8B model is the practical fine-tuning path.

InternVL's MIT license is the most permissive in VLM open-weight AI. No restrictions beyond attribution and liability disclaimer — full commercial use, redistribution, modification permitted. This makes InternVL the safest legal choice for commercial VLM deployment. Contrast with [LLaVA](/families/llava) (Apache 2.0, still very permissive) and [Gemma 3](/families/gemma) (Gemma Terms with specific restrictions). Dynamic resolution tile processing is InternVL's key deployment feature and complexity. The model processes up to 40 tiles of 448×448 for a single image, each tile encoded independently through the vision encoder. Configure tile count based on use case: 1 tile for quick visual chat, 12-20 tiles for document OCR, 40 tiles for high-resolution chart/table analysis. The vision encoder's 6B parameter size creates a latency profile unlike other VLMs. At 6B parameters, the InternViT encoder is as large as many text-only LLMs. Each image encoding pass requires a full forward pass through 6B parameters. On [RTX 4090](/hardware/rtx-4090), single-tile encoding: ~1.5s. On CPU (llama.cpp), single-tile encoding: ~15s — making CPU-only InternVL deployment impractical. Chat template: InternVL uses a specific multimodal chat format with `<|im_start|>` / `<|im_end|>` tokens and `<image>` placeholder tokens. The image placeholder is replaced with visual features by the model infrastructure. Using text-only chat templates or incorrect image placeholder formatting produces garbage output. Multi-image support: InternVL 2.5 supports up to 32 images per conversation, interleaved with text. Each image's visual tokens are inserted at the `<image>` position. This enables complex visual reasoning tasks across multiple documents but the VRAM cost scales with total visual tokens across all images. The InternVL model ecosystem on ModelScope vs HuggingFace: model weights are available on both platforms, but Chinese-language documentation, deployment scripts, and fine-tuning guides are more complete on ModelScope. For English-language deployments, HuggingFace provides adequate documentation but with less depth than ModelScope. A known deployment issue on vLLM: InternVL 2.5's pixel shuffle compression (reducing visual tokens by 4×) is not supported in vLLM versions before 0.6.3. Earlier versions produce 4× more visual tokens, increasing VRAM consumption and reducing throughput at high tile counts. Verify vLLM version >= 0.6.3 for InternVL 2.5.