Video
pose transfer
motion retargeting

Motion Transfer

Applying motion from a source video to a target subject — pose-driven dance generation, lip-sync, gesture transfer.

Capability notes

Motion transfer extracts movement patterns from a source video and applies them to a target image or character — the target moves like the source while retaining its own visual identity. Three approaches exist in 2026: AnimateDiff (diffusion-based, most flexible), MotionCtrl (camera + object motion separated), and Move AI (commercial markerless mocap). The open-weight landscape is AnimateDiff-dominant. AnimateDiff (Guo et al., 2024) injects a learned motion module into any Stable Diffusion 1.5 checkpoint. The motion module is trained on video clips to predict per-frame noise offsets that represent motion — when inserted into an image generation pipeline, it generates a sequence of images where objects move according to the learned motion prior. Key property: motion modules are plug-and-play — swap the Stable Diffusion checkpoint and the same motion module produces different visual styles with the same movement. A [realistic vision model](/models/realistic-vision) + AnimateDiff gives photorealistic motion; an anime model + same motion module gives anime-style motion. MotionCtrl (Tencent, 2024) separates camera motion from object motion into two controllable signals: a camera motion vector (global, affects all pixels) and an object motion vector (local, affects segmented regions). This enables "camera pans right while the character walks left" — compound motion that AnimateDiff struggles with. MotionCtrl requires an additional motion-conditioning network and uses ~2 GB more VRAM than AnimateDiff for equivalent resolution. Move AI (commercial, not open-weight) uses smartphone video + markerless motion capture to extract 3D skeletal motion. Output is a BVH/FBX animation file usable in Blender, Unreal, etc. Move AI is the only path from real-human motion to 3D character animation without a motion-capture suit. Cost: $30-150/month subscription. **Quality hierarchy**: Move AI (ground-truth motion capture) > MotionCtrl (separated camera + object motion) > AnimateDiff with custom motion module (fine-tuned for specific motion) > AnimateDiff with stock motion module (pre-trained on generic web video motion). The tradeoff is control vs accessibility — Move AI requires a human actor + subscription; AnimateDiff with stock modules requires zero training. Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).

If you just want to try this

Lowest-friction path to a working setup.

Start with AnimateDiff in [ComfyUI](/tools/comfyui) using the AnimateDiff Evolved custom node. This is the easiest path to motion transfer: the motion module is pre-trained and plug-and-play. You need a Stable Diffusion 1.5 checkpoint (dreamshaper, realisticVision, anything-v5) and the AnimateDiff motion module (guoyww/animatediff-motion-adapter-v1-5-3 from Hugging Face, ~1.5 GB). Hardware requirement: 8 GB VRAM minimum. [RTX 3060 12GB](/hardware/rtx-3060-12gb) or [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb) generates 16-frame clips at 512×512 in 30-60 seconds. AnimateDiff works at lower resolutions than Wan video gen — 512×512 is standard, 768×512 is comfortable at 12 GB. Workflow in ComfyUI: Load Checkpoint → AnimateDiff Loader (motion module) → CLIP Text Encode (positive prompt describing the motion you want) → KSampler with AnimateDiff → output frames → Video Combine. The prompt controls what moves and how: "a person running in place, fluid motion, 16fps" — not "a running person" (the subject is determined by your image prompt or input image, not the motion prompt). Key insight for beginners: AnimateDiff produces motion from a prompt, not by copying motion from a source video. True "extract motion from video A, apply to image B" requires video-to-motion extraction (extract optical flow or skeletal motion, feed as conditioning) — this is MotionCtrl territory, significantly more complex. AnimateDiff gives you "generate motion matching this description" which is more accessible but less controllable. For your first test: use a simple prompt ("a gentle swaying motion, slow pan, 16fps"), generate 16 frames, watch the output. Increase frames to 32 once 16 works. Motion quality degrades beyond 32-48 frames with stock modules — expect objects to morph or fade by frame 40-50.

For production deployment

Operator-grade recommendation.

Production motion transfer requires choosing between diffusion-based motion generation (AnimateDiff/MotionCtrl) and true motion capture (Move AI) based on use case, fidelity requirements, and talent availability. **Diffusion-based pipeline** (AnimateDiff/MotionCtrl): Used for synthetic video — background animations, abstract motion graphics, stylized character animation where exact motion fidelity matters less than visual quality. Architecture: [ComfyUI](/tools/comfyui) with AnimateDiff Evolved + MotionCtrl nodes → batch generation with multiple motion modules → post-processing (frame interpolation, upscaling) → delivery. Single [RTX 4090](/hardware/rtx-4090): ~60-120 16-frame clips/hour. Use for volume content (social media loops, animated album art, website hero animations). **Motion capture pipeline** (Move AI → 3D engine): Used for character animation where motion must match human performance — game cinematics, virtual production, VTuber rigging. Architecture: smartphone camera → Move AI app (captures performer) → exports BVH/FBX → import into Blender/Unreal/Maya → retarget to 3D character rig → render. No GPU requirement for capture (runs on phone); rendering uses standard 3D pipeline GPUs ([RTX 4070](/hardware/rtx-4070)+ for real-time Unreal rendering). Performer cost: $50-200/hr for talent + $30-150/month Motion AI subscription. **Hybrid pipeline** (mocap → 3D → AnimateDiff stylization): Capture motion via Move AI → animate 3D character → render low-resolution preview → use as AnimateDiff input (image-to-video with motion conditioning) → apply artistic style. Combines mocap accuracy with AI stylistic freedom. Used for stylized cinematics where character movements must be physically accurate but visuals are painted/concept-art style. **Cost comparison**: AnimateDiff self-hosted = $0.01-0.03/clip (GPU compute only). Move AI mocap = $5-20/min of captured animation (talent + subscription + rendering). API motion transfer (Runway Motion Brush, Pika Motion) = $0.02-0.10/clip. AnimateDiff wins for cost at scale; Move AI wins for fidelity; APIs win for convenience + quality on 1-10 clips.

What breaks

Failure modes operators see in the wild.

**Motion-to-subject mismatch.** Symptom: the motion module generates running-leg movement but the subject is a car — legs animate where wheels should spin. Cause: AnimateDiff motion modules are trained on human-centric video; non-human subjects get human-motion artifacts. Mitigation: fine-tune motion module on subject-appropriate video (cars driving for vehicle motion, animals moving for creature animation), or use MotionCtrl's object-specific motion vectors. **Temporal identity collapse.** Symptom: a character's face and clothing change subtly each frame — by frame 24, they look like a different person in similar clothing. Cause: Stable Diffusion generates each frame independently; the motion module only constrains position, not appearance. Mitigation: use ControlNet (canny/depth) on the first frame, propagate as conditioning to all subsequent frames. This locks subject appearance at cost of reduced motion flexibility. **Camera motion induced from static input.** Symptom: input frame is a static character portrait, but AnimateDiff adds both subject motion + spurious camera shake. Cause: stock motion modules were trained on handheld phone video — camera wobble is in the training distribution. Mitigation: use MotionCtrl's separated camera/object motion, set camera motion vector to zero, or use a motion module fine-tuned on tripod-stabilized footage. **Background warping during foreground motion.** Symptom: a character walks left, and the background stretches/distorts to follow. Cause: the diffusion model doesn't understand foreground/background separation — it moves all pixels toward the motion direction. Mitigation: generate foreground character with motion on transparent background, composite over static background plate. This is the standard production fix but requires 2× generation passes. **Motion magnitude miscalibration.** Symptom: a prompt requesting "subtle breathing motion" produces wild full-body sway. Cause: the motion module's magnitude scale doesn't align with language — "subtle" has no calibrated mapping to pixel displacement. Mitigation: use motion scale parameter (AnimateDiff's `motion_scale` typically 0.5-1.5), iterate with small increments (0.1 steps) until magnitude matches intent. Different modules have different scale baselines.

Hardware guidance

**Minimum viable (8 GB VRAM)**: [RTX 3060 12GB](/hardware/rtx-3060-12gb) or [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb). AnimateDiff with SD 1.5 at 512×512, 16 frames, FP16 — fits 8-10 GB. 30-60s per clip. [Intel Arc A770 16GB](/hardware/intel-arc-a770-16gb) via DirectML in ComfyUI: 45-90s per clip, functional but slower. The sweet spot for casual AnimateDiff use. **Recommended (12-16 GB)**: [RTX 4070 Ti 12GB](/hardware/rtx-4070-ti) or [RTX 5070 Ti 16GB](/hardware/rtx-5070-ti). AnimateDiff at 768×512, 32 frames. 20-40s per clip. Also runs MotionCtrl which adds ~2 GB VRAM overhead over AnimateDiff for equivalent settings. **Production (24 GB+)**: [RTX 4090 24GB](/hardware/rtx-4090) or [RTX 3090 24GB](/hardware/rtx-3090). AnimateDiff at 1024×576, 48 frames, FP16. MotionCtrl with full camera + object motion separation at 768×512. 15-30s per clip. [Apple M3 Ultra 64GB](/hardware/apple-m3-ultra) runs AnimateDiff via MLX at 512×512, 2-3× slower than NVIDIA but viable. **3D rendering hardware** (Move AI pipeline): Any GPU that runs Blender/Unreal. [RTX 4070 12GB](/hardware/rtx-4070) for Blender Eevee rendering, [RTX 4090 24GB](/hardware/rtx-4090) for Unreal Lumen path tracing at real-time. Rendering cost dominates: a 60s character animation at 30fps = 1,800 frames, 5-15 min/frame for path-traced rendering = 150-450 GPU-hours. Use render farms for production. **Multi-GPU**: AnimateDiff doesn't support tensor parallelism — one generation per GPU. Scale horizontally with multiple ComfyUI workers. 4× [RTX 4090](/hardware/rtx-4090) = 4 concurrent AnimateDiff clips, ~240-480 clips/hour.

Runtime guidance

**If you want the easiest AnimateDiff workflow** → [ComfyUI](/tools/comfyui) with AnimateDiff Evolved custom node. Standard and only well-supported runtime. Workflow: Load SD 1.5 checkpoint → AnimateDiff Loader (motion module) → prompt → KSampler → Video Combine. Handles motion module loading, frame scheduling, and output assembly in one graph. **If you need camera + object motion separation** → ComfyUI with MotionCtrl nodes. MotionCtrl requires: (1) source video for motion extraction, (2) target image for visual style, (3) camera motion extractor (estimates global motion), (4) object motion extractor (estimates local motion). The workflow is more complex: Load video → MotionCtrl Camera Extract → MotionCtrl Object Extract → apply to target image via KSampler. For programmatic use: ComfyUI API mode with parameterizable MotionCtrl workflows. **If you need true motion capture for 3D characters** → Move AI (commercial) on iPhone/Android → export BVH/FBX → Blender retargeting add-on (Auto-Rig Pro or Rokoko) → Unreal Engine/Blender for rendering. No pure open-weight pipeline exists for mocap-quality motion transfer. The closest open-weight alternative is VideoPose3D (Meta) for 3D pose estimation from video, which produces skeleton data you can retarget — but it's pose estimation, not motion capture, with 3-5cm joint error vs sub-cm for Move AI. **If building a motion library for reuse** → Batch AnimateDiff with motion prompt catalog. Generate 50-100 motion clips with standardized prompts ("walk_left_16fps," "idle_sway_24fps," "run_sprint_16fps") → quality filter (clip motion score via optical flow magnitude) → save as reusable MP4 loops. Build a motion library over time — each new generation can reference offline motion modules. **What doesn't work for motion transfer**: [Automatic1111](/tools/automatic1111) (image-only, no AnimateDiff support in A1111 — use ComfyUI for any video/animation), [Ollama](/tools/ollama)/[LM Studio](/tools/lm-studio) (LLM-focused, no diffusion video), [Diffusers](/tools/diffusers) (AnimateDiff works but ComfyUI's node graph is more production-stable for multi-step animation pipelines).

Setup walkthrough

  1. Install ComfyUI via Stability Matrix.
  2. ComfyUI Manager → Install Models → search "animate-diff" or "mimicmotion" for pose-driven animation.
  3. For pose-driven motion transfer (dance, gestures):

Approach 1 — AnimateDiff + ControlNet:

  • Load a reference image (the character you want to animate)
  • Load a driving video (the motion source — could be a dance video)
  • Extract pose from driving video with DWPose (ComfyUI node)
  • Feed pose sequence to AnimateDiff + ControlNet → character moves with the driving pose
  • 16 frames (~1 second) in 30-60 seconds on 12 GB GPU

Approach 2 — MimicMotion (more recent, better quality):

  • pip install mimicmotion → feed reference image + driving video → outputs animated character
  • Better temporal consistency than AnimateDiff
  1. First motion-transferred clip in 1-5 minutes per second of output.

The cheap setup

Used RTX 3060 12 GB ($200-250, see /hardware/rtx-3060-12gb). Runs AnimateDiff + ControlNet (pose) at 30-60 seconds per 16-frame clip (1 second of animation at 16 fps). For a 3-second dance clip: ~2-3 minutes. Pair with Ryzen 5 5600 + 32 GB DDR4 + 1TB NVMe. Total: ~$390-440. Motion transfer is lighter than full video generation — AnimateDiff is an SD 1.5-based method, only needs 4-6 GB VRAM. At $400, you can transfer motion to characters reliably at near-real-time for short clips.

The serious setup

Used RTX 3090 24 GB (~$700-900, see /hardware/rtx-3090). Runs AnimateDiff + multiple ControlNets (pose + depth) at 20-40 seconds per 16-frame clip. For longer sequences (5-10 seconds), the extra VRAM prevents the temporal artifacts that occur when AnimateDiff runs out of context memory. For SDXL-based animation (AnimateDiff-XL): 24 GB handles 32-frame sequences smoothly. Total: ~$1,800-2,200. Motion transfer is significantly lighter than text-to-video generation — even 8 GB cards handle basic workflows.

Common beginner mistake

The mistake: Using a driving video of a professional dancer doing complex full-body spins and jumps, expecting a static portrait photo to animate with the same motion. Why it fails: The motion model sees a source video with extreme pose changes (arms behind back, crouching, jumping) and a reference image in a neutral standing pose. The model can't map extreme poses to an image that lacks the corresponding body parts visible. The legs disappear behind the body — the model hallucinates limbs. The fix: Match the driving video to the reference image. If your reference is a standing portrait, use a driving video of someone nodding, talking, or making small gestures. If you need complex dance motion, the reference image should show the full body in a neutral dance pose. The model maps pose-to-pose — if the driving pose has limbs in positions not visible in the reference, you get artifacts. Garbage in, garbage out applies doubly to motion transfer.

Recommended setup for motion transfer

Recommended runtimes

Browse all tools for runtimes that fit this workload.

Budget build
AI PC under $1,000 →
Best GPU for this task
Best GPU for Stable Diffusion + image gen →

Reality check

Local video gen is genuinely possible in 2026 (LTX-Video, Mochi) but VRAM-hungry. 24 GB is the working minimum; 32 GB is the comfort zone for long-form workflows. Below 24 GB, video gen isn't realistic with current models.

Common mistakes

  • Trying video gen on 16 GB cards (model + KV cache doesn't fit)
  • Underestimating runtime VRAM (peak draw 1.5x model size on long sequences)
  • Mixing video gen with concurrent LLM serving on same GPU
  • Using Mac Silicon for video gen — viable but 30-50% slower than CUDA

What breaks first

The errors most operators hit when running motion transfer locally. Each links to a diagnose+fix walkthrough.

Before you buy

Verify your specific hardware can handle motion transfer before committing money.

Buyer guides
Compare hardware
Troubleshooting
Specialized buyer guides
Updated 2026 roundup