InternVL 2.5 78B
InternVL 2.5 flagship. Approaches frontier proprietary VLMs on document and OCR tasks.
Positioning
InternVL 2.5 78B is the flagship dense vision-language model from OpenGVLab, released under the permissive MIT license. With 78 billion parameters and a 32,768-token context window, it is designed for datacenter deployment and targets document understanding and OCR tasks at a level approaching frontier proprietary VLMs. Its MIT license makes it one of the most permissively licensed high-capacity VLMs available, suitable for commercial use without restriction.
Strengths
- Permissive MIT license: Unlike many large VLMs that use non-commercial or custom licenses, InternVL 2.5 78B is fully open for commercial deployment, fine-tuning, and redistribution.
- Dense architecture: As a dense 78B model, it dedicates all parameters to every forward pass, which can provide consistent quality across tasks without the routing complexity of mixture-of-experts models.
- Large context window: 32,768 tokens of context enables processing of long documents, multi-page PDFs, or high-resolution images with detailed captions.
- Designed for document and OCR tasks: The model is specifically optimized for vision-language tasks involving text-heavy images, making it a strong candidate for enterprise document processing pipelines.
Limitations
- High hardware requirements: At FP16, the model requires 156 GB of disk space, and even at Q4_K_M (43.9 GB) the memory footprint with KV cache and framework overhead can exceed 60 GB, placing it firmly in datacenter territory.
- No community benchmarks available: We do not yet have independent, community-run benchmark results for this model. Operators should treat vendor-published metrics as best-case and verify performance on their own data.
- Dense 78B inference cost: Unlike MoE models that activate only a subset of parameters, this dense model uses all 78B parameters per token, resulting in higher compute and memory demands per inference step.
- Limited deployment flexibility: The model cannot run on consumer or workstation GPUs (single 24 GB or dual 24 GB) even with aggressive quantization; it requires multi-GPU datacenter hardware (e.g., A100 80 GB or H100 nodes).
What it takes to run this locally
InternVL 2.5 78B is a datacenter-class model. Quantized sizes range from 156 GB (FP16) down to ~25.4 GB (Q2_K), but practical deployment requires adding 30–50% overhead for KV cache and framework memory. For example, Q4_K_M at ~43.9 GB plus overhead may exceed 60 GB, necessitating multiple high-memory GPUs. Even the smallest Q2_K quant (25.4 GB) with overhead may require a single 48 GB GPU (e.g., A6000) but with limited context. Realistic deployment uses multi-GPU setups (e.g., 2–4 A100 80 GB) with FP16 or Q8_0 for full quality.
Should you run this locally?
Yes if you need a permissively licensed, high-capacity VLM for document and OCR tasks in a datacenter environment, and you have access to multi-GPU hardware (A100/H100 nodes). The MIT license allows unrestricted commercial use, making it suitable for proprietary document processing pipelines.
No if you lack datacenter-grade GPUs, need low-latency single-GPU inference, or require a model that can run on consumer hardware. Also, if your tasks are general-purpose vision-language (e.g., captioning, VQA on natural images), smaller models may be more practical.
Catalog cross-links
- InternVL 2.5 8B
- InternVL 2.5 26B
- Qwen2-VL 72B
Overview
InternVL 2.5 flagship. Approaches frontier proprietary VLMs on document and OCR tasks.
How to run it
InternVL 2.5 78B is OpenGVLab's multimodal model — 78B text backbone with a vision encoder based on InternViT. Run at Q4_K_M via llama.cpp with llava-server or vLLM multimodal pipeline. Q4_K_M file size ~45 GB (text) + ~4-6 GB (vision). Minimum VRAM: 48 GB — RTX A6000 at Q3_K_M with vision, or text-only Q4_K_M. Recommended: A100 80GB at AWQ-INT4 for full vision serving. Throughput: ~8-15 tok/s on A6000 at Q4_K_M text-only; vision encoding adds ~2-4s per image. InternVL uses a custom architecture (InternViT + InternLM2/LLaMA backbone) — ecosystem support is narrower than Llama-based vision models. Check llama.cpp InternVL support before provisioning. Ollama may not have InternVL 2.5 — use raw llama.cpp llava-server. For production serving: vLLM with custom model registration (if supported). InternVL is known for strong vision-language benchmarks, especially on document understanding and OCR-heavy tasks.
Hardware guidance
Minimum: RTX A6000 48GB at Q3_K_M + vision (tight). Recommended: A100 80GB at AWQ-INT4. VRAM math: 78B dense at Q4_K_M ≈ 45 GB. InternViT encoder: ~5-8 GB (varies by resolution). KV cache at 8K: ~12 GB. Total with vision: ~62-65 GB. Single A6000 48GB is 15+ GB short — must use Q3_K_M or text-only Q4_K_M. Dual RTX 3090 48 GB total: Q4_K_M text-only or Q3_K_M + vision. A100 80GB: comfortable for Q4 + vision + 8K. Mac Studio M4 Ultra 128GB: Q4_K_M + vision, 2-5 tok/s (Apple Silicon InternVL support uncertain). Cloud: A100 at $5-10/hr. InternVL's InternViT is large — expect 2-3× the vision encoder VRAM of Llama 3.2 Vision's CLIP encoder.
What breaks first
- InternVL architecture support. llama.cpp's InternVL support is experimental — vision features may not project correctly, causing garbled image descriptions. Validate against reference outputs from the official InternVL GitHub repo. 2. InternViT VRAM bloat. The InternViT encoder is 6B+ parameters — significantly larger than typical vision encoders (CLIP is ~300M). At high resolutions, InternViT activations can spike to 10-15 GB. 3. Tokenizer incompatibility. InternVL may use a different tokenizer than standard LLaMA. Using the wrong tokenizer silently produces incorrect image token embeddings. 4. Multimodal GGUF availability. Pre-converted multimodal GGUFs for InternVL are less common than Llama 3.2 Vision. You may need to convert from hf yourself.
Runtime recommendation
Common beginner mistakes
Mistake: Using a Llama 3.2 Vision mmproj with InternVL text GGUF. Fix: Vision projectors are architecture-specific. Download the InternVL mmproj from the InternVL hf repo. Mistake: Assuming InternVL works with standard Ollama vision tags. Fix: InternVL requires custom model registration. Use llama.cpp directly with the correct multimodal GGUF. Mistake: Sending high-res images expecting InternViT to handle them. Fix: InternViT is large but has fixed input resolution limits. Resize images to the encoder's expected size to avoid OOM. Mistake: Expecting InternVL to run at Llama 3.2 Vision's VRAM footprint. Fix: InternViT is 5-10× larger than CLIP. Vision VRAM is proportionally higher. Budget extra 5-10 GB for InternViT.
Family & lineage
How this model relates to others in its lineage. Family members share architecture and training-data roots; parent / children edges record direct distillation or fine-tune relationships.
Strengths
- MIT license
- Frontier-tier OCR
Weaknesses
- 48GB+ VRAM tier
Quantization variants
Each quantization trades model quality for file size and VRAM. Q4_K_M is the most popular starting point.
| Quantization | File size | VRAM required |
|---|---|---|
| Q4_K_M | 45.0 GB | 52 GB |
Get the model
HuggingFace
Original weights
Source repository — direct quantization required.
Hardware that runs this
Cards with enough VRAM for at least one quantization of InternVL 2.5 78B.
Models worth comparing
Same parameter band, plus what's one tier above and below — so you can decide what actually fits your hardware.
Frequently asked
What's the minimum VRAM to run InternVL 2.5 78B?
Can I use InternVL 2.5 78B commercially?
What's the context length of InternVL 2.5 78B?
Does InternVL 2.5 78B support images?
Source: huggingface.co/OpenGVLab/InternVL2_5-78B
Reviewed by RunLocalAI Editorial. See our editorial policy for how we research and verify model claims.
Related — keep moving
Verify InternVL 2.5 78B runs on your specific hardware before committing money.