Inpainting

Local inpainting — filling masked regions of an image with contextually-appropriate content — is served by two dominant open-weight families in 2026: Flux Fill (Black Forest Labs) and SDXL Inpainting (Stability AI). Quality has advanced from blurry fill (2023 SD 1.5 inpainting) to photorealistic content-aware generation indistinguishable from original image content in ~80% of cases. Flux Fill (Flux.1-Fill-dev, 12B params) leads open-weight inpainting quality. Operates in two modes: (1) outpaint-compatible generation where the model fills masked regions using surrounding pixel context and an optional text prompt; (2) prompt-driven inpainting where the text description guides what appears in the masked area. Flux Fill's MMDiT architecture processes the entire image — masked and unmasked regions — jointly, producing globally coherent output. Text prompt adherence for inpainting: "add a red car" produces a red car in the masked region with correct perspective, lighting, and shadows matching the original scene in ~70% of generations. Object removal quality (masking unwanted object, no text prompt): removes object and fills background realistically in ~85% of cases — occasional blur artifacts at mask boundaries. SDXL Inpainting (2.6B+ params) is the more accessible option. Specialized inpainting checkpoint fine-tuned from SDXL base with mask-conditioned UNet. 8 GB VRAM minimum. Object removal quality: ~65–75% realistic fill — lower than Flux Fill, with visible transition artifacts around mask boundaries in ~25–30% of generations. Strength: enormous LoRA ecosystem allows style-specific inpainting — "fill with anime style," "fill with watercolor texture" — specialized styles Flux Fill's prompt-conditioning cannot replicate without a dedicated LoRA. Brush-based vs mask-based workflows. Brush tools (GIMP heal, Photoshop content-aware fill) use non-AI algorithms (PatchMatch, Poisson blending) — fast (100–500ms), deterministic, but limited to texture synthesis. Mask-based inpainting (draw a mask region, model fills) handles structural content (entire objects, complex backgrounds) that brush tools cannot. Hybrid: brush for small blemishes (sensor dust, skin spots), mask-based for object removal and structural edits. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b). Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).

Use [ComfyUI](/tools/comfyui) for local inpainting. Install via Stability Matrix (stabilitymatrix.com) — one-click installer that handles Python, CUDA, and dependency resolution. Flow for object removal: 1. Open ComfyUI (localhost:8188), load the Flux Fill workflow from ComfyUI's built-in workflow library. 2. Download Flux.1-Fill-dev model (~23 GB) via ComfyUI Manager → Install Models → search "flux1-fill-dev". 3. Load your image in a Load Image node. 4. Right-click the image → Open in MaskEditor → paint over object to remove with brush → Save. 5. Connect mask output to Flux Fill's mask input. 6. Keep prompt empty for object removal (model infers context from surrounding pixels) or type what to fill with ("wooden floor", "brick wall"). 7. Set steps=20–30, guidance=3.5, Queue Prompt. On [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb): 5–10 seconds per inpaint at 1024×1024. [RTX 4090](/hardware/rtx-4090): 2–4 seconds. [RTX 3060 12GB](/hardware/rtx-3060-12gb) with Flux Fill FP8 via ComfyUI-GGUF: 8–15 seconds — VRAM at floor, expect occasional OOM on large images. If Flux Fill exceeds VRAM, use SDXL Inpainting: download "sd_xl_base_1.0_inpainting.safetensors" via ComfyUI Manager. SDXL inpaints at 4–8 seconds on [RTX 3060 12GB](/hardware/rtx-3060-12gb). Quality is lower — expect ~20% more visible mask boundaries — but 8 GB is the VRAM floor. Alternative quick setup: [Automatic1111 WebUI](/tools/automatic1111) with SDXL Inpainting model. Install A1111, download SDXL inpainting checkpoint, switch to img2img → Inpaint tab → draw mask → generate. Simpler UI than ComfyUI but limited to SDXL inpainting (no Flux Fill support).

Production inpainting pipelines optimize for quality-confidence per image and throughput. **Two-stage architecture.** Stage 1: fast pre-check — analyze mask region, classify difficulty. Small masks (<5% image area), uniform backgrounds (sky, wall, grass): SDXL Inpainting with low steps (10–15), 2–4 seconds on [L40S](/hardware/nvidia-l40s). Large masks (>20% area), complex backgrounds (crowded scenes, textured surfaces): Flux Fill at 20–30 steps, 4–8 seconds. Classification via simple heuristics (mask area ratio, background texture variance via Laplacian variance). Routes 60–70% of inpainting requests through fast path. **Quality assurance.** Post-generation quality check: (1) mask boundary analysis — compute gradient magnitude at mask edges, high gradient = visible transition artifact, flag for human review; (2) content consistency — CLIP similarity between inpainted region and surrounding context, below threshold = fill doesn't match scene; (3) structural continuity — edge detection on inpainted image, check that edges crossing mask boundary align (e.g. horizon line continues straight). Images failing QA (15–25%) re-route for re-generation with adjusted parameters (higher steps, different seed). **Throughput and cost.** [RTX 4090](/hardware/rtx-4090): Flux Fill 2–4 seconds per inpaint → 15–30/minute. SDXL Inpainting 1–2 seconds → 30–60/minute. [L40S](/hardware/nvidia-l40s) at batch=4: Flux Fill 6–12 seconds for 4 concurrent → 20–40 effective/minute. At 100K inpainting operations/month, local L40S costs ~$400/month rental. Compare to cloud API (Clipdrop/Replicate): $0.01–0.05/inpaint → $1,000–5,000/month. Local ROI: 1–3 months payback. **Batch processing for e-commerce.** Product photo cleanup: remove backgrounds, remove blemishes, standardize lighting. Pipeline: (1) automatic mask generation — SAM 2 segments product from background, generate foreground/background masks; (2) background inpainting — Flux Fill replaces old background with studio-white or custom backdrop; (3) blemish removal — detect dust/scratches via difference-of-Gaussian blob detector, create small masks, SDXL inpaint. Throughput on [L40S](/hardware/nvidia-l40s): 100–200 products/hour. Human review step samples 5% of outputs for quality monitoring. **High-resolution strategy.** Inpainting at 4K: VRAM exceeds 24 GB. Strategy: (1) downsample 4K → 1024, inpaint at low res, (2) upscale inpainted result to 4K using Real-ESRGAN, (3) blend high-res edges: overlay upscaled inpainted region onto original 4K with Poisson blending at mask boundaries. Quality: 90% of full-resolution inpaint quality at 25% VRAM cost. Accept minor blur at mask edges for products beyond 1024 native generation.

- **Output quality degradation under load.** Symptom: response quality drops under high concurrency — outputs become shorter and less detailed. Cause: [vLLM](/tools/vllm) under GPU memory pressure reduces effective context or switches generation parameters. Mitigation: monitor per-request quality scores, set max concurrency based on memory headroom, scale horizontally when metrics degrade. - **Confidence hallucination.** Symptom: outputs are confident and well-structured but factually wrong. Cause: LLMs generate by probability not factual verification. Confident-sounding outputs are harder to detect as errors. Mitigation: implement deterministic validation on outputs (numeric bounds, schema checks). Flag for human review when validation fails. Never trust LLM self-reported confidence. - **Edge case collapse.** Symptom: model performs well on common Inpainting tasks (90%+) but fails catastrophically on rare edge cases. Cause: training distribution under-represents edge cases; model defaults to nearest common pattern. Mitigation: build regression test suite of edge cases, run before every model update, track edge case pass rate separately from general accuracy. - **Context pollution in long sessions.** Symptom: after 20-30 exchanges the model mixes information from earlier conversation parts, producing contradictions. Cause: long contexts dilute attention — relevant context competes with noise. Mitigation: periodically summarize and reset context every 15-20 exchanges, store key findings externally, trigger summarization at 80% context window capacity. For models see [Llama 3.3 70B](/models/llama-3-3-70b).

**Hobbyist tier ($300–600).** [RTX 3060 12GB](/hardware/rtx-3060-12gb): SDXL Inpainting at 4–8 seconds per inpaint at 1024×1024. Flux Fill FP8/NF4 via GGUF at 8–15 seconds — functional but at VRAM floor. Expect OOM on high-resolution inputs (>1536×1536) with Flux Fill. [RX 7600 XT](/hardware/rx-7600-xt) at 16 GB: SDXL Inpainting at 6–12 seconds — AMD ROCm for ComfyUI requires manual configuration, expect 20–40% slower than NVIDIA. [Intel Arc B580](/hardware/intel-arc-b580) at 12 GB: SDXL Inpainting at 8–15 seconds via Intel IPEX — functional but smallest ComfyUI extension ecosystem. **Enthusiast tier ($1,500–2,500).** [RTX 4090](/hardware/rtx-4090) at 24 GB: the inpainting sweet spot. Flux Fill at 2–4 seconds, SDXL Inpainting at 1–2 seconds. Fits Flux Fill + 1 ControlNet (depth/canny) + LoRA. [RTX 5090](/hardware/rtx-5090) at 32 GB: Flux Fill + 2 ControlNets + 2 LoRAs simultaneously — enables complex inpainting pipelines (depth-conditioned + style-transfer in one pass). [RTX 5080](/hardware/rtx-5080) at 16 GB: Flux Fill at 3–6 seconds, SDXL at 1.5–3 seconds — but 16 GB restrictive for Flux + ControlNet combos. [RX 7900 XTX](/hardware/rx-7900-xtx) at 24 GB: SDXL Inpainting at 3–6 seconds via ROCm on Linux — AMD's ComfyUI support for Flux Fill is unstable as of mid-2026. **SMB tier ($6,000–15,000).** [RTX 6000 Ada](/hardware/rtx-6000-ada) at 48 GB: Flux Fill + 3 ControlNets + 5 LoRAs + 2048×2048 output simultaneously. Batch=4 Flux Fill at 8–15 seconds. [L40S](/hardware/nvidia-l40s) at 48 GB: identical for 24/7 production inpainting — e-commerce, real estate, media asset pipelines. **Enterprise tier ($25,000+).** [A100 80GB SXM](/hardware/nvidia-a100-80gb-sxm): Flux Fill at full precision with batch=8. All ControlNet combos, highest-resolution output (4096×4096). Overkill for inpainting-only but justified if shared with [text-to-image](/tasks/text-to-image) and [outpainting](/tasks/outpainting) workloads. [L40S](/hardware/nvidia-l40s) at 48 GB is more cost-effective for dedicated inpainting. **VRAM vs quality.** More VRAM enables (in priority): higher resolution output > multiple ControlNets > larger batch size > faster generation. 24 GB floor for professional Flux Fill; 16 GB floor for SDXL-only; 12 GB floor for SDXL with offloading.

**ComfyUI vs Automatic1111 vs Diffusers — inpainting workflow comparison.** [ComfyUI](/tools/comfyui) is the production standard for inpainting in 2026. Node-based workflow: Load Image → Load Mask (or MaskEditor) → Load Checkpoint (Flux Fill/SDXL) → VAE Encode (image + mask) → KSampler → VAE Decode → Save Image. MaskEditor node provides interactive brush/eraser tool directly in ComfyUI. Drag-and-drop mask images from external editors (Photoshop, GIMP) via Load Image node. Custom nodes: "Inpaint Crop and Stitch" crops around mask region, inpaints only the crop, stitches back — reduces VRAM 60–80% for small masks on large images. "Segment Anything" node auto-generates masks by clicking — SAM 2 integration. [Automatic1111 WebUI](/tools/automatic1111) (and SD WebUI Forge) uses linear workflow: img2img tab → Inpaint sub-tab → upload image → draw mask → select model → generate. Simpler for casual use. Forge fork (by lllyasviel) provides better memory management — 30–40% lower VRAM for SDXL. Limitation: SDXL Inpainting only — Flux Fill not supported. Acceptable for hobbyist SDXL-based inpainting; not production-grade due to lack of batch processing and API mode limitations. [Diffusers](/tools/diffusers) for programmatic inpainting. Hugging Face pipeline: `StableDiffusionXLInpaintPipeline` for SDXL, `FluxFillPipeline` for Flux Fill. Python API: load image + mask (PIL Image), call pipeline, output PIL Image. Use for: (1) API-level control over generation parameters (step schedule, prompt weighting, latent manipulation), (2) integration into larger image processing pipelines (e-commerce workflow, photo app backend), (3) custom sampling schedules and post-processing. Tradeoffs: manual GPU memory management, no visual workflow debugging, requires Python coding. **Decision tree.** Interactive editing (designer/retoucher): ComfyUI with MaskEditor. Automated pipeline (e-commerce, batch processing): Diffusers + Python. Simple casual use (<10 images/week): Automatic1111 Forge. Max quality with hardware constraints: ComfyUI + Inpaint Crop and Stitch + Flux Fill FP8/NF4. Professional tool integration (Photoshop/GIMP roundtrip): export mask from editor as PNG, import to ComfyUI, inpaint, export result back to editor for final touch-up.