Vision
bbox detection
yolo
detection

Object Detection

Locating and labeling specific objects in images with bounding boxes. Specialized detection models (YOLO family, DETR) dominate, though VLMs increasingly handle simple detection via prompting.

Capability notes

Local object detection in 2026 spans three architectural families with different speed–accuracy–generality tradeoffs. Specialized detection models (YOLO family) dominate throughput-critical use cases. Open-vocabulary detectors (Grounding DINO, OWLv2) handle novel object categories without retraining. Multimodal LLMs are emerging as detection-capable but at 10–50× higher compute cost. YOLO-world (YOLOv8-world, Apache 2.0) achieves open-vocabulary detection at 20–60 FPS on [RTX 4090](/hardware/rtx-4090). Trained on detection datasets with text conditioning, it detects any object describable in natural language — "person wearing a red jacket," "car with open trunk" — without fine-tuning. LVIS benchmark (1203 categories): 35.4 mAP — below fine-tuned detectors (45–52 mAP) but game-changing for variable label sets. For fixed-category detection (security cameras detecting person/car/bicycle), fine-tuned YOLOv8-nano runs at 200–300 FPS on [RTX 3060 12GB](/hardware/rtx-3060-12gb) with 50–60 mAP on COCO. Grounding DINO (IDEA Research, Apache 2.0) combines DINO detection architecture with text grounding — given an image and a phrase ("the red cup on the table"), outputs bounding box + confidence. COCO zero-shot: 52.5 mAP. Superior to YOLO-world on compositional queries (spatial relationships, attributes) because DINO's transformer backbone processes global image context. Speed: 2–8 FPS on consumer GPUs — 5–10× slower than YOLO-world but 15–20% more accurate on complex scenes. OWLv2 (Google, Apache 2.0) scales open-vocabulary detection to 2B parameters with a ViT-g backbone. LVIS zero-shot: 43.2 mAP. Best open-weight zero-shot accuracy for long-tail categories (rare objects, specialized equipment). Speed: 0.5–2 FPS — reserved for batch processing where accuracy outweighs throughput. OWLv2's training on WebLI (10B image-text pairs) gives it the broadest visual vocabulary — detects "antique brass doorknob" or "industrial conveyor belt roller" where other detectors fail. Multimodal LLMs (Qwen2.5-VL, InternVL2) can output bounding boxes when prompted: "Draw bounding boxes around all people in this image." Accuracy: 48–55 mAP on COCO — below specialized detectors. Strength: combined detection + description — "there are 3 people, one is waving, one is holding a coffee cup." Use VLM detection when the task requires both localization and semantic reasoning.

If you just want to try this

Lowest-friction path to a working setup.

Install [Ollama](/tools/ollama) with a vision model for zero-setup object detection: ```bash ollama pull llama3.2-vision:11b-instruct-q4_K_M ``` Prompt for detection with bounding boxes: ```python import requests, base64 with open("scene.jpg", "rb") as f: img = base64.b64encode(f.read()).decode() resp = requests.post("http://localhost:11434/api/generate", json={ "model": "llama3.2-vision:11b-instruct-q4_K_M", "prompt": "Detect all objects in this image. For each object, output " "the object name and approximate bounding box as [x1, y1, x2, y2] " "normalized to 0-1000 coordinates.", "images": [img], "stream": False }) print(resp.json()["response"]) ``` VLM detection works for occasional use but is slow (2–5 seconds/image) and approximate (boxes ±15% accurate). For real-time or batch detection, use [Hugging Face Transformers](/tools/transformers) with a dedicated model: ```python from transformers import pipeline detector = pipeline("object-detection", model="ultralytics/YOLO-world-detr") results = detector("scene.jpg") for r in results: print(f"{r['label']}: {r['score']:.2f} at {r['box']}") ``` YOLO-world-v2-s runs at 15–30 FPS on [RTX 3060 12GB](/hardware/rtx-3060-12gb). For fixed-category detection (common objects only), use standard YOLOv8-nano — 120–200 FPS on the same GPU. Install via `pip install ultralytics` for the simplest dedicated detector setup. On [Apple Silicon](/hardware/apple-m4-pro), use YOLOv8-nano CoreML export for 8–15 FPS on M4 Pro.

For production deployment

Operator-grade recommendation.

Production object detection requires matching model architecture to throughput, latency, and accuracy requirements. **Fixed-category (security, manufacturing, retail).** Fine-tuned YOLOv8-m on your labeled data. Training: 2,000–10,000 labeled images, 50–200 epochs, 2–8 hours on [RTX 4090](/hardware/rtx-4090). Inference: 50–100 FPS at 640×640 input. mAP: 55–75 depending on category complexity. Deploy via ONNX Runtime: export YOLO to ONNX, serve with Triton Inference Server on [L4](/hardware/nvidia-l4) — 200–400 FPS at batch=8. Cost per 1M frames on L4 (~$0.40/hr): $0.30. **Open-vocabulary (content moderation, visual search, retail cataloging).** Grounding DINO-T (172M params) for real-time (<100ms): COCO zero-shot 48.5 mAP at 8–15 FPS on [L40S](/hardware/nvidia-l40s). Grounding DINO-B (341M params) for balanced speed/accuracy: 52.5 mAP at 4–8 FPS. OWLv2-B/16 (2B params) for batch accuracy-first: 43.2 LVIS mAP at 0.5–2 FPS — process offline batch of 10,000 images in 1.5–5 hours on [A100 80GB](/hardware/nvidia-a100-80gb-sxm). **Hybrid pipeline: YOLO + Grounding DINO.** Stage 1: YOLOv8-nano (high recall, low precision, 200 FPS) identifies candidate objects with permissive threshold. Stage 2: Grounding DINO-T refines candidates, classifies, filters false positives. End-to-end throughput: 20–40 FPS. Accuracy: 5–12 mAP gain over YOLO-only while maintaining 10× throughput over Grounding DINO-only. Use this pattern when your label set is 80% fixed categories + 20% variable descriptions. **Video detection optimization.** Frame-by-frame detection is wasteful — objects move slowly between frames (30 FPS = 33ms). Detection strategy: (1) detect on keyframes (every 5th frame), (2) track detected objects between keyframes using ByteTrack or BoT-SORT (CPU-only, <1ms/frame), (3) re-detect on tracked frames only when tracking confidence drops below threshold. Reduces GPU inference load 70–85% while maintaining per-frame bounding boxes. **Cost analysis — cloud API vs local.** AWS Rekognition: $0.001/image (first 1M). At 1M images/month = $1,000 API cost. Local on [L4](/hardware/nvidia-l4) (~$300/month rental): $0.0004/image (including GPU rental). Break-even at 300K images/month. At 10M images/month, local saves $9,000/month.

What breaks

Failure modes operators see in the wild.

**Open-vocabulary false positives.** Symptom: Grounding DINO detects "person" on a mannequin, "car" on a toy car, "dog" on a statue. Cause: open-vocabulary detectors match visual patterns without understanding scale, context, or object affordances. A mannequin and a person share visual features in embedding space. Mitigation: add scale priors (expected object size at given distance), use depth estimation to filter out-of-scale detections, implement hard-negative mining during post-processing — "toy car" vs "real car" is an embedding-space distinction that few-shot fine-tuning can learn. **Occlusion and truncation.** Symptom: person behind a desk — YOLO detects "person" at 0.95 confidence, fails to detect "person" when 60% occluded (confidence drops to 0.21). Cause: detectors learn full-object appearance — occluded objects produce partial feature vectors that don't match training distribution. Mitigation: training data augmentation with random occlusion during training improves occluded detection by 8–15 mAP. At inference, lower detection threshold for edge-of-frame objects (likely truncated) and cascade to high-res crop for verification. **Small object blindness.** Symptom: surveillance camera captures a scene where a distant person is 15×15 pixels — YOLO misses it entirely. Cause: detector backbones downsample input 32× — objects smaller than 32×32 pixels in feature space become a single activation, indistinguishable from noise. Mitigation: tiled inference — split 4K image into overlapping 1024×1024 tiles, detect per-tile, NMS-merge across tiles. Upsample input to detector (640×640 → 1280×1280) at 4× VRAM cost. Specialized small-object detectors (TinyPerson-trained YOLO, SAHI framework) improve small object mAP by 15–25. **Class confusion on fine-grained categories.** Symptom: detector confuses "golden retriever" for "labrador retriever" — both have similar visual appearance. Cause: detector's classification head operates on coarse features — no fine-grained discriminative features at classification layer depth. Mitigation: two-stage classification — detector outputs "dog" (coarse), crop detection region, classify crop with fine-grained classifier (DINOv2 + linear probe). Adds 10–30ms per detection but improves fine-grained accuracy 20–35%.

Hardware guidance

**Hobbyist tier ($300–600).** [RTX 3060 12GB](/hardware/rtx-3060-12gb): YOLOv8-nano at 120–200 FPS, YOLO-world-v2-s at 10–18 FPS. Fits all YOLO variants and Grounding DINO-T. Cannot fit OWLv2 (requires 16 GB+). [Intel Arc B580](/hardware/intel-arc-b580) at 12 GB: YOLOv8 via OpenVINO at 80–140 FPS — functional but 30–40% slower than NVIDIA equivalent. [RX 7600 XT](/hardware/rx-7600-xt) at 16 GB: YOLOv8 via ONNX Runtime DirectML at 60–100 FPS — adequate for 30 FPS video detection. **Enthusiast tier ($1,500–2,500).** [RTX 4090](/hardware/rtx-4090) at 24 GB: YOLOv8-m at 60–100 FPS, Grounding DINO-B at 4–8 FPS, YOLO-world-v2-l at 15–25 FPS. Fits all detectors including OWLv2-B/16 at FP16. The detection workhorse — handles real-time video detection (30 FPS YOLO) + batch open-vocabulary (Grounding DINO overnight). [RTX 5090](/hardware/rtx-5090) at 32 GB: same profile with 15–25% throughput gain from increased memory bandwidth. **SMB tier ($6,000–15,000).** [RTX 6000 Ada](/hardware/rtx-6000-ada) at 48 GB: batch processing at scale — YOLOv8 batch=32 at 300–500 effective FPS, Grounding DINO batch=8 at 15–30 effective FPS. Fits OWLv2-L/14 at FP16 for high-accuracy batch detection. [L40S](/hardware/nvidia-l40s) at 48 GB: datacenter equivalent, better sustained thermals for 24/7 CCTV processing. **Enterprise tier ($25,000+).** [A100 80GB SXM](/hardware/nvidia-a100-80gb-sxm): all models at FP16 with batch headroom. YOLOv8-x batch=64 at 500–800 effective FPS. OWLv2-L/14 at 2–4 FPS batch=8. [H100 PCIe](/hardware/nvidia-h100-pcie): limited benefit for detection — models are bandwidth-bound, not compute-bound, and H100 has same 2.0 TB/s bandwidth as A100. **Single-board / edge.** Jetson Orin Nano (8 GB): YOLOv8-nano at 25–40 FPS via TensorRT. Jetson Orin AGX (64 GB): YOLOv8-m at 40–60 FPS, Grounding DINO-T at 2–4 FPS. Raspberry Pi 5 + Hailo-8L accelerator: YOLOv8-nano at 25–30 FPS for fixed-category detection. Coral Edge TPU: MobileDet at 15–25 FPS — limited to COCO categories.

Runtime guidance

**Ultralytics vs Transformers vs TensorRT — detection runtime selection by throughput and deployment environment.** Ultralytics YOLO (`pip install ultralytics`) is the default detection framework. Unified API for training, validation, inference, and export. Loads YOLOv5/v8/v9/v10/v11, YOLO-world, and YOLO-NAS. Inference API: `model("image.jpg")` returns list of detection results. Export to ONNX, TensorRT, CoreML, TFLite. Tradeoff: Python overhead limits maximum FPS — 200 FPS on [RTX 4090](/hardware/rtx-4090) in Ultralytics becomes 350+ FPS via TensorRT. [Hugging Face Transformers](/tools/transformers) + AutoModelForObjectDetection loads DETR-family, Grounding DINO, OWLv2, RT-DETR. Standardized API across architectures — swap "yolos-tiny" for "grounding-dino-tiny" by changing model ID string. Best for prototyping and multi-model comparisons. Tradeoff: 2–3× slower than framework-native inference due to HF pipeline overhead and lack of architecture-specific optimizations. TensorRT (NVIDIA) compiles YOLO models to optimized CUDA engine. YOLOv8-nano TensorRT on [RTX 4090](/hardware/rtx-4090): 350–450 FPS vs 200 FPS in Ultralytics Python. Export workflow: Ultralytics → ONNX → trtexec → TensorRT engine. Pay the 5–10 minute compilation cost once per model; 50–80% throughput gain at inference. Essential for high-throughput production (CCTV, manufacturing line inspection). **Serving framework selection.** Single camera → Ultralytics Python loop (simplest). Multi-camera (4–16 streams) → [DeepStream](/tools/tensorrt-llm) (NVIDIA) handles multi-stream decode + inference + tracking in one GPU pipeline. REST API serving → Triton Inference Server with ONNX or TensorRT backend — dynamic batching accumulates requests over 5–10ms window for batch inference. CPU deployment → ONNX Runtime with OpenVINO execution provider on Intel CPUs (YOLOv8-nano at 8–15 FPS on i7-13700K). **Decision tree.** <1 camera, real-time: Ultralytics Python. 1–4 cameras, real-time: TensorRT + Python multiprocessing. 4–16 cameras, real-time: DeepStream on single GPU. Batch processing (<10K images/day): Ultralytics or Transformers. Batch processing (>10K images/day): Triton + TensorRT. CPU-only or edge: ONNX Runtime + OpenVINO/ARM NN. Apple Silicon: CoreML export via Ultralytics for Metal-accelerated inference.

Setup walkthrough

  1. pip install ultralytics (YOLO — the standard open-weight detection library).
  2. For zero-shot detection (no training): pip install transformers → use OWLv2:
from transformers import pipeline
detector = pipeline("zero-shot-object-detection", model="google/owlv2-base-patch16")
results = detector("living_room.jpg", candidate_labels=["sofa", "lamp", "tv", "plant"])
for r in results:
    print(f"{r['label']}: {r['score']:.2f} at {r['box']}")
  1. First detection in <1 second on GPU, 2-5 seconds on CPU.
  2. For custom classes with training: yolo detect train data=custom_dataset.yaml model=yolo11n.pt epochs=50 — trains in 10-30 minutes on 100 labeled images.
  3. For real-time detection: YOLO11n processes 200+ fps on RTX 3060, 30+ fps on CPU.
  4. For video: YOLO processes frame-by-frame with tracking (BoT-SORT/DeepSort built-in).

The cheap setup

Used GTX 1060 6 GB (~$60). Runs YOLO11n at 100-150 fps — real-time detection for multiple camera feeds. OWLv2 at 5-10 fps for zero-shot detection. Train a custom YOLO11s on 500 images in ~20 minutes. Pair with Ryzen 5 5600 + 16 GB DDR4 + 512 GB NVMe. Total: ~$320-370. For CPU-only: YOLO11n runs at 25-35 fps on a modern laptop CPU — viable for single-camera real-time. Object detection is incredibly GPU-efficient — even integrated graphics handle real-time.

The serious setup

Used RTX 3060 12 GB (~$200-250, see /hardware/rtx-3060-12gb) is the sweet spot — runs YOLO11x (largest variant) at 80-120 fps. Handles 8+ 1080p camera streams simultaneously with tracking. Trains custom YOLO11x on 10K images in 1-2 hours. For production surveillance systems (16+ cameras): add a second RTX 3060. Total build: ~$700-900. For Jetson edge deployment: see /hardware/jetson-ai. Object detection is one of the most mature and GPU-efficient AI tasks — prioritize camera quality and storage, not GPU.

Common beginner mistake

The mistake: Using a multimodal LLM (GPT-4V, Qwen2-VL) for object detection on a video stream, wondering why it processes 1 frame every 5 seconds. Why it fails: VLMs generate text — they describe what they see in natural language. Object detection requires numerical output (bbox coordinates, class IDs) at 30+ fps. A VLM doing detection is like using a novelist to read a spreadsheet. The fix: Use a dedicated detection model (YOLO, DETR, OWLv2 for zero-shot). These output bounding boxes + class scores directly as tensors. YOLO11n processes 200+ fps vs. 0.2 fps for a VLM. Use VLMs only for the 0.1% of detections where you need reasoning ("Is this person holding a weapon or a phone?"). Detection is a throughput problem — use the right tool for throughput.

Recommended setup for object detection

Recommended runtimes

Browse all tools for runtimes that fit this workload.

Budget build
AI PC under $1,000 →
Best GPU for this task
Best GPU for local AI →

Reality check

Local AI workloads have real hardware constraints that vary by task type. VRAM ceiling decides what model fits; bandwidth decides decode speed; compute decides prefill speed. Pick the GPU tier that fits your actual workload, not the spec sheet.

Common mistakes

  • Buying for spec-sheet VRAM without modeling KV cache + activation overhead
  • Underestimating quantization quality loss below Q4
  • Skipping flash-attention support (real perf gap on long context)
  • Ignoring sustained-load thermals (laptops thermal-throttle within 30 min)

What breaks first

The errors most operators hit when running object detection locally. Each links to a diagnose+fix walkthrough.

Before you buy

Verify your specific hardware can handle object detection before committing money.

Buyer guides
Compare hardware
Troubleshooting
Specialized buyer guides
Updated 2026 roundup