Vision
semantic segmentation
sam
instance segmentation
Image Segmentation
Pixel-level region labeling — semantic, instance, or panoptic segmentation. Specialized models (SAM family, Mask2Former) dominate. Critical for medical imaging, robotics, content creation.
Capability notes
Image segmentation — pixel-level labeling of image regions — is dominated by the Segment Anything Model (SAM) family in 2026. SAM 2 (Meta, Apache 2.0) extends the original SAM architecture with video segmentation and improved interactive segmentation quality. The local deployment landscape splits into three modes: interactive (user clicks → mask), automatic (model segments everything), and promptable (text/box → mask).
SAM 2 Hiera-L (224M params) produces masks in 15–40ms on [RTX 4090](/hardware/rtx-4090) for interactive segmentation — click on an object, get a mask in less than one frame. Automatic segmentation (segment-everything mode): SAM 2 generates 50–200 masks per image in 2–8 seconds depending on image resolution and complexity. SA-1B dataset-trained models generalize to objects never seen in training — a chair, a dog, a satellite antenna, a microscope slide — without fine-tuning. This zero-shot generalization is SAM's core value proposition.
EfficientSAM and MobileSAM distill SAM's capabilities into smaller models. MobileSAM (10M params, 35 MB) segments at 5–10ms on mobile GPUs — 10–20× faster than SAM-H at 60–75% of mask quality (mIoU on SA-1B validation). EfficientSAM-Ti (10M params) runs at 20–40ms on CPU — viable for web-based segmentation via Transformers.js.
Specialized segmentation models (Mask2Former, OneFormer) dominate fixed-domain tasks. Mask2Former with Swin-L backbone achieves 57.8 PQ on COCO panoptic — 15–20 points above SAM's universal segmentation on the same benchmark. For medical imaging (cell segmentation, organ delineation), U-Net variants fine-tuned on domain data achieve 90%+ Dice scores where SAM's zero-shot falls below 60%.
Multimodal LLMs offer text-prompted segmentation: "segment the person wearing a blue shirt." Qwen2.5-VL and InternVL2 generate bounding boxes which SAM refines into masks. This VLM → SAM pipeline combines open-vocabulary understanding with pixel-precise output — the dominant approach for "segment the [complex description]" queries where no dedicated text-to-mask model exists. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b).
If you just want to try this
Lowest-friction path to a working setup.
Use SAM 2 via [Hugging Face Transformers](/tools/transformers) for interactive segmentation. Install:
```bash
pip install transformers torch opencv-python matplotlib
```
Load SAM 2 and segment by clicking on an image:
```python
from transformers import pipeline
import cv2
# Segment-everything (automatic)
segmenter = pipeline("mask-generation", model="facebook/sam2-hiera-large")
img = cv2.imread("photo.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
results = segmenter(img, points_per_side=32)
# results["masks"] is list of binary masks
# results["scores"] is confidence per mask
for i, mask in enumerate(results["masks"]):
print(f"Mask {i}: area={mask.sum()}px, score={results['scores'][i]:.3f}")
```
For interactive segmentation (click on object):
```python
from transformers import SamModel, SamProcessor
model = SamModel.from_pretrained("facebook/sam2-hiera-large")
processor = SamProcessor.from_pretrained("facebook/sam2-hiera-large")
inputs = processor(img, input_points=[[[500, 300]]], return_tensors="pt")
outputs = model(**inputs)
masks = processor.image_processor.post_process_masks(
outputs.pred_masks, inputs["original_sizes"], inputs["reshaped_input_sizes"]
)
```
SAM 2 Hiera-L runs on [RTX 3060 12GB](/hardware/rtx-3060-12gb) at 2–8 seconds for automatic segmentation. For lighter hardware, use MobileSAM: `pipeline("mask-generation", model="vzhong/MobileSAM")` — 0.5–2 seconds at lower mask quality on the same GPU.
For non-technical use, many image editors (GIMP with SAM plugin, Photoshop's "Object Selection" tool) now integrate SAM-like segmentation without writing code.
For production deployment
Operator-grade recommendation.
Production segmentation pipelines select model size and inference mode based on accuracy, latency, and throughput requirements.
**Interactive segmentation (human-in-the-loop).** SAM 2 Hiera-L serves web-based annotation tools. Client sends click coordinates → server runs SAM → returns mask in <50ms end-to-end. Target: <100ms total (network + inference) to feel responsive. On [L40S](/hardware/nvidia-l40s), SAM 2 Hiera-L at batch=1: 15–25ms inference. With 20ms network roundtrip, total = 35–45ms — responsive. Throughput: 20–40 requests/second on single GPU. Deploy via FastAPI + ONNX Runtime with CUDA execution provider.
**Automatic segmentation (batch processing).** For e-commerce catalog processing (segment products from backgrounds), SAM 2 runs in automatic mode generating 50–150 masks per image. Post-processing: filter masks by predicted IoU score (>0.88 keeps 80% of high-quality masks), merge overlapping masks (NMS with mask IoU >0.7), classify each mask with CLIP to label the segmented object. Pipeline throughput on [RTX 6000 Ada](/hardware/rtx-6000-ada): 100–200 images/hour at 1024×1024. Cost per 10,000 product images: ~$3–5 GPU time.
**Video segmentation.** SAM 2's video mode propagates masks across frames. Give mask on frame 0 → SAM 2 predicts masks on frames 1–N without re-prompting. Propagation speed: 20–40ms per frame on [A100 80GB](/hardware/nvidia-a100-80gb-sxm). Quality degrades: after 50 frames, mask IoU drops to 75–80% (from 95% on frame 1) — re-prompt every 30–50 frames to maintain quality. Use case: video object removal, background replacement, rotoscoping.
**Medical imaging integration.** SAM 2 as initialization for specialized models: SAM generates candidate masks, domain-specific model (nnU-Net fine-tuned on 500 labeled medical images) refines masks to clinical accuracy. SAM alone: 55–65% Dice on unseen medical domains. SAM + nnU-Net refinement: 85–92% Dice. The refinement model is small (5–20M params, <500 MB) and runs on CPU — inference time 100–500ms per image.
**Cost vs cloud API.** Manual annotation (human): $0.10–1.00 per mask depending on complexity. SAM-assisted annotation (human clicks, SAM draws): $0.02–0.10 per mask in human time. Fully automatic SAM: $0.0001–0.001 per mask in GPU compute. For annotating 100K images with 20 objects each = 2M masks: human-only = $200K–2M; SAM-assisted = $2K–20K; fully automatic = $200–2,000.
What breaks
Failure modes operators see in the wild.
**Over-segmentation.** Symptom: SAM generates 300+ masks for a simple scene with 10 objects — most masks are fragments of larger objects. Cause: SAM's automatic mode uses a grid of prompt points — each point generates a mask, many points land on sub-regions of the same object producing overlapping fragments. Mitigation: increase `pred_iou_thresh` to 0.95 (keeps only highest-quality masks, reduces count 60%), apply mask NMS with IoU threshold 0.7 to merge overlapping masks, use stability scoring (masks stable across threshold variation are more likely to be true objects).
**Boundary bleeding.** Symptom: hair segmentation — mask includes background visible through hair strands, or conversely cuts into the hair outline. Cause: SAM operates on downsampled features (16× or 32×) — fine structures below 16px are lost in feature space. Mitigation: use SAM's high-resolution mask refinement (upsamples mask to input resolution with learned refinement), apply matting post-processing (alpha matting for hair/transparent objects), use SAM 2's multi-mask output mode and select mask with best boundary alignment.
**Domain shift failure.** Symptom: SAM achieves 92% IoU on natural images (COCO-style), 55% on medical CT scans, 40% on satellite imagery. Cause: SAM trained on SA-1B (11M natural images) — medical, satellite, microscopy, and industrial inspection images are outside training distribution. Mitigation: fine-tune SAM on 500–2,000 domain images using LoRA (low-rank adaptation) — brings domain accuracy to 75–85% IoU. Full fine-tuning of SAM decoder on 10K+ images achieves 85–95% IoU.
**Text-prompt misalignment.** Symptom: user prompts "segment the red car" — VLM correctly identifies car but SAM segments wrong car (or car + shadow + reflection). Cause: VLM outputs bounding box, SAM segments whatever occupies that box — including occluders, shadows, reflections. The VLM→box→SAM pipeline has two failure points: VLM box accuracy and SAM mask precision within the box. Mitigation: use SAM's point-prompt mode with points inside and outside the target object rather than box mode, iterate with negative points (click background to exclude).
Hardware guidance
**Hobbyist tier ($300–600).** [RTX 3060 12GB](/hardware/rtx-3060-12gb): SAM 2 Hiera-B+ (base) at 2–6 seconds auto-segmentation per image. MobileSAM at 0.3–1 second. Interactive segmentation: 20–40ms per click — feels responsive. 12 GB fits all SAM variants except Hiera-H (huge) which needs 16 GB at FP32. [Intel Arc B580](/hardware/intel-arc-b580) at 12 GB: SAM via OpenVINO at 3–8 seconds — functional but expect setup friction (ONNX export + OpenVINO config). [Apple M4 Pro](/hardware/apple-m4-pro) with 24 GB unified memory: SAM 2 Hiera-L via CoreML at 2–5 seconds.
**Enthusiast tier ($1,500–2,500).** [RTX 4090](/hardware/rtx-4090) at 24 GB: SAM 2 Hiera-L at 1.5–3 seconds auto-segmentation. Interactive: 10–20ms — zero perceptible latency. Fits all SAM variants including Hiera-H for maximum accuracy. [RTX 5090](/hardware/rtx-5090) at 32 GB: same profile with 20–30% faster image encoding.
**SMB tier ($6,000–15,000).** [RTX 6000 Ada](/hardware/rtx-6000-ada) at 48 GB: SAM 2 Hiera-H at 2–4 seconds. 48 GB handles 4K×4K images without tiling. Batch processing: 4 concurrent inferences for auto-segmentation throughput. [L40S](/hardware/nvidia-l40s) at 48 GB: identical profile — datacenter pick for annotation serving.
**Enterprise tier ($25,000+).** [A100 80GB SXM](/hardware/nvidia-a100-80gb-sxm): all SAM variants at FP16 with batch headroom. SAM 2 video mode processes 30–50 FPS for real-time video segmentation. 80 GB fits image encoder + decoder + KV cache for video memory (stores frame features for mask propagation). [H200](/hardware/nvidia-h200) at 141 GB: SAM 2 video mode at full resolution (1080p) with 300+ frame memory — processes 10-second clips without re-encoding.
**Cloud GPU cost guide.** Spot instance [L4](/hardware/nvidia-l4) at $0.25/hr: 50–100 images/hr SAM auto-segmentation = $0.003–0.005/image. Spot [L40S](/hardware/nvidia-l40s) at $0.80/hr: 200–400 images/hr = $0.002–0.004/image. On-demand [A100 80GB](/hardware/nvidia-a100-80gb-sxm): $1.50–2.50/hr, 300–600 images/hr = $0.003–0.008/image. Image batch processing economics strongly favor local or spot instances — 24/7 operation recovers hardware cost in 3–6 months vs on-demand.
Runtime guidance
**Transformers vs ONNX vs CoreML — SAM 2 deployment paths.**
[Hugging Face Transformers](/tools/transformers) provides the standard Python API for SAM 2: `SamModel.from_pretrained()` loads model, `model(**inputs)` generates masks. Supports interactive (point/box prompts) and automatic (segment-everything) modes. Best for research, prototyping, and sub-1,000 images/day. Tradeoff: Python overhead limits maximum throughput — 2–8 seconds per auto-segmentation on [RTX 4090](/hardware/rtx-4090) is adequate for batch processing but not real-time video.
ONNX Runtime with CUDA execution provider for production serving. Export SAM 2 to ONNX: `torch.onnx.export()` on the encoder and decoder separately. Encoder runs once per image and caches features; decoder runs per click/mask. ONNX decoder at 5–10ms per click on [L40S](/hardware/nvidia-l40s) — 2–3× faster than PyTorch due to fused operations and lower Python overhead. Serve via Triton Inference Server for multi-client annotation platforms.
CoreML (Apple) for native Mac/iOS segmentation. Export SAM 2 to CoreML via `coremltools`. On [M4 Max](/hardware/macbook-pro-16-m4-max), CoreML SAM 2 at 1.5–3 seconds auto-segmentation — comparable to PyTorch on NVIDIA. Advantage: zero-copy between CPU and GPU on unified memory, Metal Performance Shaders (MPS) acceleration, no CUDA dependency. Best path for macOS-native segmentation apps.
**Video segmentation with SAM 2.** SAM 2 video mode operates differently: encode first frame → decoder generates mask for frame 0 → memory attention propagates mask to frame 1 → decoder refines → repeat. The memory bank stores encoded features from past frames and mask predictions — grows linearly with video length (50–100 MB per frame for Hiera-L). After 200+ frames, memory pruning is required — SAM 2 drops oldest frames from memory, reducing quality on frames that re-enter the scene after long absence.
**Annotation tool integration.** SAM 2 serves as backend for annotation platforms: CVAT, Label Studio, Supervisely. The platform sends click coordinates → server runs SAM → returns mask polygon → annotator adjusts → resubmits. Annotation throughput: manual = 50–100 objects/hour; SAM-assisted = 200–500 objects/hour (4–5× improvement). For training data generation at scale (100K images), SAM-assisted annotation reduces cost from $10K–100K to $2K–25K.
Setup walkthrough
pip install segment-anything(Meta's SAM 2 — SOTA open-weight segmentation).- Download the model (auto on first use, ~150 MB for SAM 2.1 tiny, ~2.4 GB for SAM 2.1 large).
For automatic segmentation (SAM 2 auto):
from sam2 import SAM2ImagePredictor
predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-large")
predictor.set_image("photo.jpg")
masks = predictor.generate() # segments every object in the image
for i, mask in enumerate(masks):
mask_np = mask.cpu().numpy() # binary mask array
cv2.imwrite(f"mask_{i}.png", (mask_np * 255).astype('uint8'))
- First segmentation in 2-5 seconds on GPU, 10-30 seconds on CPU.
- For interactive: paint a point or box → SAM segments the object at that location.
- For video segmentation: SAM 2 video model propagates masks across frames with minimal re-prompting.
The cheap setup
SAM 2.1 Tiny (150 MB) runs on CPU at 5-15 seconds per image — practical for batch processing of 100s of images overnight. A used GTX 1060 6 GB ($60) runs SAM 2.1 Large at 2-5 seconds per image. For video segmentation (SAM 2 video): GTX 1660 Super 6 GB (~$100) handles 720p video at 5-10 fps. Pair with Ryzen 5 5600 + 16 GB DDR4 + 512 GB NVMe. Total: ~$320-370. Segmentation is moderate compute — the models are small but the mask operations are spatially expensive.
The serious setup
Used RTX 3060 12 GB (~$200-250, see /hardware/rtx-3060-12gb). Runs SAM 2.1 Large at 1-2 seconds per image, SAM 2 video at 15-30 fps for 1080p. Can segment 10K+ images/hour in batch. For medical imaging (3D segmentation): 12 GB handles typical CT/MRI volumes (512×512×200 voxels) with sliding window inference. Total build: ~$700-900. For very large 3D volumes (whole-body CT, 1000+ slices): 24 GB GPU recommended. Segmentation is VRAM-light for 2D, VRAM-hungry for 3D.
Common beginner mistake
The mistake: Running SAM in "automatic everything" mode on a complex scene with 50+ objects, then spending hours manually sorting through 50 masks to find the one you want. Why it fails: SAM's automatic mode segments everything — background clutter, shadows, reflections. You get 50 masks when you only needed 3. The fix: Use prompt-based segmentation. Give SAM a single point or bounding box on the object you want: predictor.predict(point_coords=[[x, y]], point_labels=[1]). SAM segments exactly that object. For batch processing similar images, use SAM-assisted labeling: segment 10 images with prompts, fine-tune a smaller model (YOLO-seg, Mask R-CNN) on those masks, then run the fine-tuned model on 10K images at 100× speed. SAM is a labeling tool, not a classifier.
Recommended setup for image segmentation
Recommended hardware
Best GPU for local AI →
All workloads ranked across VRAM tiers.
Recommended runtimes
Browse all tools for runtimes that fit this workload.
Reality check
Local AI workloads have real hardware constraints that vary by task type. VRAM ceiling decides what model fits; bandwidth decides decode speed; compute decides prefill speed. Pick the GPU tier that fits your actual workload, not the spec sheet.
Common mistakes
- Buying for spec-sheet VRAM without modeling KV cache + activation overhead
- Underestimating quantization quality loss below Q4
- Skipping flash-attention support (real perf gap on long context)
- Ignoring sustained-load thermals (laptops thermal-throttle within 30 min)
What breaks first
The errors most operators hit when running image segmentation locally. Each links to a diagnose+fix walkthrough.
Before you buy
Verify your specific hardware can handle image segmentation before committing money.
Buyer guides
Compare hardware
Specialized buyer guides
Updated 2026 roundup