ComfyUI Video Pipeline

Orchestrates video generation across three engines, selecting the best one based on requirements and available resources.

Engine Selection

VIDEO REQUEST

    |

    |-- Need film-level quality?

    |   |-- Yes + 24GB+ VRAM → Wan 2.2 MoE 14B

    |   |-- Yes + 8GB VRAM → Wan 2.2 1.3B

    |

    |-- Need long video (>10 seconds)?

    |   |-- Yes → FramePack (60 seconds on 6GB)

    |

    |-- Need fast iteration?

    |   |-- Yes → AnimateDiff Lightning (4-8 steps)

    |

    |-- Need camera/motion control?

    |   |-- Yes → AnimateDiff V3 + Motion LoRAs

    |

    |-- Need first+last frame control?

    |   |-- Yes → Wan 2.2 MoE (exclusive feature)

    |

    |-- Default → Wan 2.2 (best general quality)

Pipeline 1: Wan 2.2 MoE (Highest Quality)

Image-to-Video

Prerequisites:

• wan2.1_i2v_720p_14b_bf16.safetensors in models/diffusion_models/

• umt5_xxl_fp8_e4m3fn_scaled.safetensors in models/clip/

• open_clip_vit_h_14.safetensors in models/clip_vision/

• wan_2.1_vae.safetensors in models/vae/

Settings:

Parameter

Value

Notes

Resolution

1280x720 (landscape) or 720x1280 (portrait)

Native training resolution

Frames

81 (~5 seconds at 16fps)

Multiples of 4 + 1

Steps

30-50

Higher = better quality

CFG

5-7

Sampler

uni_pc

Recommended for Wan

Scheduler

normal

Frame count guide:

Duration

Frames (16fps)

1 second

17

3 seconds

49

5 seconds

81

10 seconds

161

VRAM optimization:

• FP8 quantization: halves VRAM with minimal quality loss

• SageAttention: faster attention computation

• Reduce frames if OOM

Text-to-Video

Same as I2V but uses wan2.1_t2v_14b_bf16.safetensors and EmptySD3LatentImage instead of image conditioning.

First+Last Frame Control (Wan 2.2 Exclusive)

Wan 2.2 MoE allows specifying both the first and last frame, enabling precise video planning:

• Generate two hero images with consistent character

• Use first as start frame, second as end frame

• Wan interpolates the motion between them

Pipeline 2: FramePack (Long Videos, Low VRAM)

Key Innovation

VRAM usage is invariant to video length - generates 60-second videos at 30fps on just 6GB VRAM.

How it works:

• Dynamic context compression: 1536 markers for key frames, 192 for transitions

• Bidirectional memory with reverse generation prevents drift

• Frame-by-frame generation with context window

Settings

Parameter

Value

Notes

Resolution

640x384 to 1280x720

Depends on VRAM

Duration

Up to 60 seconds

VRAM-invariant

Quality

High (comparable to Wan)

Uses same base models

When to Use

• Videos longer than 10 seconds

• Limited VRAM systems (but RTX 5090 doesn't need this)

• When VRAM is needed for parallel operations

• Batch video generation

Pipeline 3: AnimateDiff V3 (Fast, Controllable)

Strengths

• Motion LoRAs for camera control (pan, zoom, tilt, roll)

• Effect LoRAs (shatter, smoke, explosion, liquid)

• Sliding context window for infinite length

• Very fast with Lightning model (4-8 steps)

Settings

Parameter

Value (Standard)

Value (Lightning)

Motion Module

v3_sd15_mm.ckpt

animatediff_lightning_4step.safetensors

Steps

20-25

4-8

CFG

7-8

1.5-2.0

Sampler

euler_ancestral

lcm

Resolution

512x512

512x512

Context Length

16

16

Context Overlap

4

4

Camera Motion LoRAs

LoRA

Motion

v2_lora_ZoomIn

Camera zooms in

v2_lora_ZoomOut

Camera zooms out

v2_lora_PanLeft

Camera pans left

v2_lora_PanRight

Camera pans right

v2_lora_TiltUp

Camera tilts up

v2_lora_TiltDown

Camera tilts down

v2_lora_RollingClockwise

Camera rolls clockwise

Post-Processing Pipeline

After any video generation:

1. Frame Interpolation (RIFE)

Doubles or quadruples frame count for smoother motion:

Input (16fps) → RIFE 2x → Output (32fps)

Input (16fps) → RIFE 4x → Output (64fps)

Use rife47 or rife49 model.

2. Face Enhancement (if character video)

Apply FaceDetailer to each frame:

• denoise: 0.3-0.4 (lower than image - preserves temporal consistency)

• guide_size: 384 (speed optimization for video)

• detection_model: face_yolov8m.pt

3. Deflicker (if needed)

Reduces temporal inconsistencies between frames.

4. Color Correction

Maintain consistent color grading across frames.

5. Video Combine

Final output via VHS Video Combine:

frame_rate: 16 (native) or 24/30 (after interpolation)

format: "video/h264-mp4"

crf: 19 (high quality) to 23 (smaller file)

Talking Head Pipeline

Complete pipeline for character dialogue:

1. Generate audio → comfyui-voice-pipeline

2. Generate base video → This skill (Wan I2V or AnimateDiff)

   - Prompt: "{character}, talking naturally, slight head movement"

   - Duration: match audio length

3. Apply lip-sync → Wav2Lip or LatentSync

4. Enhance faces → FaceDetailer + CodeFormer

5. Final output → video-assembly

Quality Checklist

Before marking video as complete:

• Character identity consistent across frames

• No flickering or temporal artifacts

• Motion looks natural (not jerky or frozen)

• Face enhancement applied if character video

• Frame rate is smooth (24+ fps for delivery)

• Audio synced (if talking head)

• Resolution matches delivery target

Reference

• references/workflows.md - Workflow templates for Wan and AnimateDiff

• references/models.md - Video model download links

• references/research-log.md - Latest video generation advances

• state/inventory.json - Available video models