Vision

visual q&a

image qa

image questions

Visual Question Answering

Answering natural-language questions about image contents. Modern VLMs make this accessible — Qwen2.5-VL, InternVL, LLaVA all credible.

Capability notes

Visual Question Answering (VQA) — answering natural-language questions about image content — is the canonical multimodal LLM benchmark and a practical local AI capability in 2026. The VLM processes image + text question → generates answer. Accuracy is measured on VQAv2 (natural images, free-form questions) and TextVQA (scene text reading + reasoning). VQAv2 benchmark: human accuracy 80.2%. GPT-4V (cloud): 77.2%. InternVL2-76B: 74.8% — leads open-weight. Qwen2.5-VL 72B: 73.1%. [Llama 3.2 Vision](/models/llama-3-3-70b) 90B: 71.5%. The 7–13B VLMs score 58–65% — adequate for simple questions ("what color is the car?") but fail on compositional questions ("why is the person holding an umbrella despite clear skies?"). The accuracy cliff between 13B and 70B+ VLMs is ~12–15 points — steeper than text-only LLMs because vision-language reasoning adds a modality integration challenge. VQA accuracy varies by question type. Object presence ("is there a cat?"): 92–96% across all scales. Counting ("how many people?"): 68–82% — accuracy degrades above 5 objects. Attribute identification ("what color is the coat?"): 78–88% — struggles with fine-grained colors (burgundy vs maroon). Spatial reasoning ("what is to the left of the sofa?"): 62–76%. Commonsense reasoning ("why is this scene unusual?"): 52–68%. Text reading ("what does the sign say?"): 72–88% — scene text OCR remains a relative weakness. Real-world VQA deployments differ from benchmarks. User questions are domain-specific ("is this weld defective?" "what species of mushroom is this?" "what error does this dashboard indicator show?") where zero-shot VLMs achieve 55–75% accuracy. Domain adaptation via few-shot prompting or LoRA fine-tuning on 500–2,000 domain QA pairs raises accuracy to 75–90%. Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).

If you just want to try this

Lowest-friction path to a working setup.

Use [Ollama](/tools/ollama) with a vision model for zero-setup VQA: ```bash ollama pull llama3.2-vision:11b-instruct-q4_K_M ``` Ask questions about images via Python or curl: ```python import requests, base64 with open("image.jpg", "rb") as f: img = base64.b64encode(f.read()).decode() resp = requests.post("http://localhost:11434/api/generate", json={ "model": "llama3.2-vision:11b-instruct-q4_K_M", "prompt": "What is in this image? Describe in detail. " "Are there any people? What are they doing? " "What objects do you see? What is the setting?", "images": [img], "stream": False }) print(resp.json()["response"]) ``` The 11B VLM handles basic VQA at 2–4 seconds per question on [RTX 3060 12GB](/hardware/rtx-3060-12gb). For more complex questions ("explain the physics shown in this diagram"), upgrade to a 32B+ VLM. [LM Studio](/tools/lm-studio) provides a GUI for VQA: load a vision model (Pixtral, Qwen2.5-VL), upload image, type question. The conversation history enables multi-turn VQA — ask "what is this?", follow up with "what is the brand of the laptop?" and "what year was that model released?" Model retains image context across turns. For mobile VQA, install an iOS app (MLX Chat, Private LLM) that loads llama3.2-vision or SmolVLM locally on iPhone/iPad. [iPhone AI](/tasks/iphone-ai) with A18 Pro can run 3B vision models at 3–6 seconds per question, entirely on-device. For batch VQA (100+ images, same question), use [Hugging Face Transformers](/tools/transformers) with Qwen2.5-VL to batch-process images through a single prompt template.

For production deployment

Operator-grade recommendation.

Production VQA requires model selection by accuracy class and domain adaptation strategy. **Accuracy tiers.** Tier 3 (7–13B, 55–65% VQAv2): use for object identification, simple counting, single-attribute queries where occasional errors are acceptable. Deploy on [L4](/hardware/nvidia-l4) at 1–3 seconds/question, $0.25/hr GPU cost. Tier 2 (32B, 65–72%): daily-use VQA for customer-facing applications (product Q&A, property inspection). Deploy on [L40S](/hardware/nvidia-l40s) at 2–5 seconds/question. Tier 1 (72B+, 72–75%): accuracy-critical VQA (medical image Q&A, defect inspection, insurance claim assessment). Deploy on [A100 80GB](/hardware/nvidia-a100-80gb-sxm) at 1.5–3 seconds/question. **Domain adaptation.** Zero-shot VQA accuracy on specialized domains is 10–20 points below VQAv2 benchmark. For production: (1) curate 500–2,000 domain QA pairs, (2) few-shot prompt with 3–5 examples inline — gains 5–10 accuracy points but increases prompt length 3–5×, (3) LoRA fine-tune VLM on domain QA pairs — gains 12–18 accuracy points with 2–4 hours training on [RTX 4090](/hardware/rtx-4090). QLoRA (4-bit) fine-tunes 72B VLMs on 24 GB GPUs — enables domain adaptation without datacenter hardware. **Multi-image VQA.** Users increasingly ask questions across multiple images: "which of these three products has the defect?" "compare the kitchen in photo A and photo B." VLMs handle 2–4 images in a single prompt by interleaving image tokens with text. Qwen2.5-VL supports up to 16 images in context. Multi-image VQA accuracy: 60–70% on comparison tasks — below single-image because the VLM must attend across images, splitting limited context budget. **Confidence calibration.** VQA models are poorly calibrated — they report "I'm not sure" on easy questions and confidently wrong answers on hard ones. Implement confidence scoring: (1) generate 3 answers with temperature 0.7, check consistency — 3/3 agreement = high confidence, 2/3 = medium, 1/3 = low; (2) self-consistency: prompt "on a scale of 1–10, how confident are you?" — models underestimate (report 3–4 when correct 85% of time) but rank correctly; (3) out-of-distribution detection: CLIP similarity between image + question and image + answer embedding — low similarity suggests hallucination.

What breaks

Failure modes operators see in the wild.

**Yes-bias on binary questions.** Symptom: VLM answers "is there a clock in this image?" with "yes" 80% of the time regardless of ground truth. Cause: affirmative answers are more common in training data — VLMs default to "yes" when uncertain. Mitigation: ask balanced questions ("answer yes or no: is there a clock?"), calibrate with known-negative images, use ablation prompting — ask "what objects ARE present?" then check if clock is in the list rather than asking about clock directly. **Counting failures above 5.** Symptom: "how many apples?" — VLM says 4 when there are 7. Cause: VLMs count subitizing-style (1–4 recognized instantly) but above 4 requires sequential counting — vision transformer patches don't align with objects. Mitigation: use detection-then-count pipeline — Grounding DINO detects all instances of target object, count detections. Hybrid: VQA identifies object type, detector counts instances. Accuracy improvement: 85% (VLM-only counting) to 95%+ (VLM + detector). **Spatial reasoning errors.** Symptom: "what is to the left of the couch?" — VLM answers "a lamp" when the lamp is to the right. Cause: relative spatial terms (left/right/front/behind) require 3D scene understanding from 2D image — VLMs lack depth perception and viewpoint reasoning. Mitigation: rephrase spatial queries as absolute: "describe the objects on the left side of the image," add spatial markers to prompt ("number the objects left-to-right 1-5, then answer"), use grounding models (Grounding DINO) to locate objects and compute spatial relationships algorithmically rather than via VLM. **Scene text misreading.** Symptom: VLM reads "CLOSED" as "CLOSE" or "CLOSET." Cause: scene text is variable in font, orientation, lighting, and occlusion — VLMs trained on natural images don't learn robust OCR. Mitigation: pre-extract text with dedicated OCR (PaddleOCR, Tesseract) and inject extracted text into VLM prompt: "The image contains the following text: [extracted text]. Now answer the question." Improves scene text VQA accuracy 15–25%.

Hardware guidance

**Hobbyist tier ($300–600).** [RTX 3060 12GB](/hardware/rtx-3060-12gb): runs 7–11B VLMs (llama3.2-vision:11b, Pixtral 12B) at Q4 — 2–4 seconds per question. VQAv2 accuracy: 58–65%. Adequate for personal photo Q&A, simple object ID, basic scene understanding. [RX 7600 XT](/hardware/rx-7600-xt) at 16 GB: same class of models but 20–40% slower on AMD ROCm. [Intel Arc B580](/hardware/intel-arc-b580) at 12 GB: functional but smallest VLM ecosystem support — expect compilation issues. **Enthusiast tier ($1,500–2,500).** [RTX 4090](/hardware/rtx-4090) at 24 GB: Qwen2.5-VL 32B at Q4 — 2–5 seconds per question, 68–73% VQAv2. The sweet spot for personal VQA — handles compositional questions, spatial reasoning, scene text. Can LoRA fine-tune 32B for domain adaptation. [RTX 5090](/hardware/rtx-5090) at 32 GB: fits Qwen2.5-VL 32B at Q8 or 72B at Q4 — 3–7 seconds per question. Enables highest personal-tier VQA accuracy (72–75%). **SMB tier ($6,000–15,000).** [RTX 6000 Ada](/hardware/rtx-6000-ada) at 48 GB: Qwen2.5-VL 72B at FP8 — 2–4 seconds per question, 73–75% VQAv2. Handles 4–8 concurrent VQA sessions. [L40S](/hardware/nvidia-l40s) at 48 GB: identical profile, serves as VQA API endpoint for team/department use. **Enterprise tier ($25,000+).** [A100 80GB SXM](/hardware/nvidia-a100-80gb-sxm): InternVL2-76B at FP16 — 1.5–3 seconds per question, 74–76% VQAv2. 80 GB handles 15–25 concurrent VQA sessions with KV cache for conversation history. [H200](/hardware/nvidia-h200) at 141 GB: fits largest VLMs with extra context for multi-image VQA (up to 16 images) and long conversation histories (50+ turns). **Apple Silicon.** [MacBook Pro 16" M4 Max](/hardware/macbook-pro-16-m4-max) at 64 GB+: Qwen2.5-VL 32B via [MLX-LM](/tools/mlx-lm) — 4–8 seconds per question. For mobile VQA, [Apple A18 Pro](/hardware/apple-a18-pro) runs SmolVLM or llama3.2-vision:3b at 3–8 seconds on-device via CoreML. [M4 Ultra](/hardware/apple-m4-ultra) at 128 GB+: 72B VLMs at 4–10 seconds — the desktop VQA workstation.

Runtime guidance

**Ollama vs Transformers vs vLLM — VQA serving architecture.** [Ollama](/tools/ollama) with vision models is the simplest VQA deployment. Supports llama3.2-vision, llava, minicpm-v, and pixtral. API: POST image as base64 + question → receive answer. Single-stream only — one question at a time. Best for individual use, prototyping, and <100 questions/day. GPU memory management is automatic — model loads on first request, optionally stays loaded via `OLLAMA_KEEP_ALIVE`. [Hugging Face Transformers](/tools/transformers) with Qwen2.5-VL or InternVL2 for programmatic VQA. Use `model.chat()` interface for multi-turn. Batch VQA: load multiple images, iterate with same prompt template. On [RTX 6000 Ada](/hardware/rtx-6000-ada), 1,000 questions across 200 images at 3 seconds each = 50 minutes. For domain adaptation, LoRA fine-tune with [🤗 PEFT](https://github.com/huggingface/peft) — training script: load VLM, add LoRA adapters to attention layers, train on domain QA pairs, merge adapters for inference. [vLLM](/tools/vllm) with vision support (v0.6.0+) provides production VQA serving with OpenAI-compatible API. Endpoint: `POST /v1/chat/completions` with `image_url` content blocks. Continuous batching across concurrent VQA requests. Prefix caching reuses image embeddings when multiple questions target the same image. Throughput: 15–30 concurrent VQA sessions on single [A100 80GB](/hardware/nvidia-a100-80gb-sxm) at 2–4 seconds per response. **VQA pipeline architecture.** (1) Image preprocessing: validate format, resize to model-native resolution (448–980 px), detect and correct orientation. (2) Question preprocessing: classify question type (object, count, attribute, spatial, text, commonsense), inject type-specific instructions into prompt. (3) VLM inference: generate answer with structured output format. (4) Post-processing: validate answer against question type (if count question, extract number; if yes/no, extract boolean). (5) Confidence scoring: consistency check or self-evaluation. (6) Logging: store question, image, answer, confidence for offline evaluation and model improvement.

Setup walkthrough

Install Ollama → ollama pull minicpm-v (8 GB) or ollama pull llava:13b (8 GB).
Python script for visual question answering:

import ollama
with open("image.jpg", "rb") as f:
    img = f.read()
resp = ollama.chat(model="minicpm-v", messages=[{
    "role": "user",
    "content": "How many people are in this image? What are they doing? Is anyone wearing a hat?",
    "images": [img]
}])
print(resp["message"]["content"])

First answer in 5-10 seconds. Works for counting, attribute detection, spatial reasoning, and activity recognition.
For stronger VQA: ollama pull qwen2.5-vl:7b — higher accuracy on detail-oriented questions, better OCR, 128K context for multi-image Q&A.
For batch VQA (100s of images): use a local VL model with batched inference via vLLM.
For specialized VQA benchmarks (VQAv2, GQA, TextVQA): use lm-evaluation-harness with the local VL model.

The cheap setup

Used RTX 3060 12 GB (~$200-250, see /hardware/rtx-3060-12gb). Runs MiniCPM-V at 5-10 seconds per question — practical for exploring photo collections interactively. Qwen2-VL 7B at similar speed with better detail recognition. Pair with Ryzen 5 5600 + 16 GB DDR4 + 512 GB NVMe. Total: ~$360-405. For CPU-only: LLaVA 7B via llama.cpp at 30-60 seconds per question — functional for occasional use. VQA is one of the most practically useful local AI tasks — "what's in this photo?" comes up constantly.

The serious setup

Used RTX 3090 24 GB (~$700-900, see /hardware/rtx-3090). Runs Qwen2-VL 72B at 10-20 seconds per question — the strongest local VQA available. Near GPT-4V quality on detail questions, spatial reasoning, and OCR-based VQA. For production VQA applications (customer support analyzing product photos, insurance claim photo analysis), the 72B model's accuracy justifies the slower speed. Total: ~$1,800-2,200. For speed-focused VQA: Qwen2-VL 7B on RTX 4090 at 1-3 seconds per question.

Common beginner mistake

The mistake: Asking a VLM "What's in this image?" and being satisfied with the generic answer "a living room with furniture." Why it fails: Open-ended questions produce open-ended answers. The model gives the most salient answer, not the answer you need. If you're looking for a specific detail (is the window open?), you won't get it from a generic prompt. The fix: Ask specific questions: "Is the window on the left open or closed? What color is the couch? Is there a coffee table? What's on it?" Multiple specific questions get better answers than one vague one. For batch analysis, define a questionnaire template and run it on every image. VQA quality depends more on prompt specificity than model size — a good prompt on a 7B model beats a bad prompt on a 72B model.

Reality check

Local AI workloads have real hardware constraints that vary by task type. VRAM ceiling decides what model fits; bandwidth decides decode speed; compute decides prefill speed. Pick the GPU tier that fits your actual workload, not the spec sheet.

Common mistakes

Buying for spec-sheet VRAM without modeling KV cache + activation overhead
Underestimating quantization quality loss below Q4
Skipping flash-attention support (real perf gap on long context)
Ignoring sustained-load thermals (laptops thermal-throttle within 30 min)

What breaks first

The errors most operators hit when running visual question answering locally. Each links to a diagnose+fix walkthrough.

Before you buy

Verify your specific hardware can handle visual question answering before committing money.

Buyer guides

Compare hardware

Troubleshooting

Specialized buyer guides

Updated 2026 roundup

Best free local AI tools (2026) →