ThinkV2V Brings Explicit Reasoning to Instruction-Guided Video Editing
ThinkV2V introduces a reasoning-driven framework that leverages multimodal large language models (MLLMs) for complex instruction-guided video editing, outperforming larger baselines and introducing new datasets and benchmarks for evaluation.
ThinkV2V is a new research framework designed to address the limitations of current instruction-guided video editing models, especially in cases requiring causal or semantic reasoning. Unlike earlier approaches that use MLLMs mainly as semantic encoders, ThinkV2V explicitly activates the reasoning capabilities of MLLMs prior to visual generation, resulting in improved handling of complex editing tasks.
Key Innovations
- Explicit reasoning over video and instructions using MLLMs before the video generation step.
- A novel MLLM-to-DiT architecture to generate refined conditioning signals for editing.
- Progressive Curriculum Training, advancing the model from basic editing tasks to reasoning-intensive scenarios.
- Inference-Time Thinking Scaling: iterative refinement and selection of prompts to enhance reliability in complex tasks.
- Release of the ThinkV2V-150K dataset for training and ThinkV2V-Bench benchmark for evaluation of reasoning-based editing.
Performance Impact
Experimental results show that ThinkV2V's 5B-parameter DiT model achieves state-of-the-art performance on both complex and standard editing scenarios, significantly outperforming larger 10B-parameter baselines.
