Abstract
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.
Method
VideoGen-Agent formulates video generation as a multi-turn tool-use problem. Given a prompt, the agent sequentially decides whether to seek additional context, generate a video candidate, or verify and refine an intermediate result. Tools are organised into three functional groups:
1 · Augmentation
Text web search, image search, and code-based physical simulation ground generation in external knowledge and controllable priors.
2 · Generation
A family of video generators with different conditionings: T2V, I2V, motion-conditioned, and (multi-) reference-image-to-video.
3 · Verification
Open-vocabulary detection and depth estimation inspect generated videos for subject localisation, spatial relations, and geometric consistency.
Training
Starting from Qwen3-VL-8B-Instruct, we first distil teacher trajectories via supervised fine-tuning, then optimise the agent with GRPO using a category-aware hybrid reward that combines format correctness, model-as-judge video score, and tool-quality signals. This gives stable, informative gradients for multi-turn rollouts.
VABench
Procedural Knowledge (PK)
Named techniques / processes whose execution depends on specific motion details.
Single-Entity Identity (SI)
One named real-world subject; identity must be preserved across the clip.
Multi-Entity Identity (MI)
Multiple named subjects; identity preservation under interaction.
Physics Simulation (PS)
Physically governed motion under specified initial conditions: collisions, free fall, orbits.
Compositional Scene (CS)
Multi-entity composition, inter-object interaction, spatial reasoning.
Multi-Shot (MS)
Explicit time-ordered sequences with complete and correctly ordered phases across multiple shots.
Per-category results on VABench
Scores are the Gemini 3.1 Pro judge mean in [0, 100], averaged within each category. Higher is better. Overall is the unweighted mean across all six categories.
| Model | PK | SI | MI | PS | CS | MS | Overall |
|---|---|---|---|---|---|---|---|
| Open-source baselines | |||||||
| CogVideoX-5B | 38.9 | 48.3 | 37.8 | 41.6 | 50.4 | 30.8 | 41.3 |
| Mochi-1 | 41.9 | 49.6 | 37.8 | 48.4 | 55.4 | 28.3 | 43.6 |
| HunyuanVideo 13B | 37.5 | 29.3 | 17.6 | 31.6 | 31.5 | 31.1 | 29.8 |
| Wan 2.1-T2V-14B | 42.0 | 46.2 | 39.5 | 49.5 | 60.8 | 62.0 | 50.0 |
| Commercial baselines | |||||||
| Kling 3.0 | 64.4 | 62.3 | 55.2 | 56.7 | 86.5 | 66.1 | 65.2 |
| Hailuo 2.0 (MiniMax) | 63.0 | 58.6 | 52.4 | 55.1 | 86.0 | 67.6 | 63.8 |
| Seedance 1.0 Pro Fast | 55.5 | 51.4 | 44.2 | 46.2 | 74.6 | 66.8 | 56.5 |
| Seedance 2.0 Fast | 78.0 | 75.0 | 69.0 | 60.2 | 87.8 | 68.9 | 73.2 |
| Ours | |||||||
VideoGen-Agent — RL (Toolset 1)?Toolset 1 — models used
|
80.4 | 75.1 | 65.3 | 67.9 | 83.1 | 81.7 | 75.6 |
VideoGen-Agent — RL (Toolset 2)?Toolset 2 — models used
|
92.1 | 83.2 | 86.7 | 69.2 | 90.7 | 94.6 | 86.1 |
Toolset 1 is the training-time stack VideoGen-Agent was optimised against. Toolset 2 swaps in stronger generation tools at inference time without any additional agent training, demonstrating that the trained agent benefits from subsequent advances in generation tools. Hover the small ? at the top-right of each Toolset name for the exact models (with links).
Ablation study
Rows (i)–(iv) and (viii) trace the training pipeline under a fixed Toolset 1: vanilla T2V → prompt rewriting → zero-shot tool use → SFT → RL. Rows (v)–(vi) remove individual reward components; row (vii) trains one agent per task instead of a single multitask agent. Bold is the best score in each column.
| Variant | PK | SI | MI | PS | CS | MS | Overall |
|---|---|---|---|---|---|---|---|
| (i) Vanilla T2V (Seedance 1.0 Pro Fast) | 55.5 | 51.4 | 44.2 | 46.2 | 74.6 | 66.8 | 56.5 |
| (ii) Qwen3-VL-8B-Instruct + Prompt Rewriting + T2V | 57.3 | 51.5 | 45.3 | 44.6 | 77.2 | 68.1 | 57.3 |
| (iii) Qwen3-VL-8B-Instruct + System Prompt + T2V (zero-shot tool use) | 61.2 | 54.8 | 47.1 | 49.6 | 75.5 | 68.9 | 59.5 |
| (iv) VideoGen-Agent-SFT-Toolset 1 | 72.7 | 68.1 | 58.6 | 60.4 | 79.2 | 76.3 | 69.2 |
| (v) VideoGen-Agent-RL-Toolset 1 — w/o VLM reward | 76.4 | 74.3 | 63.9 | 66.5 | 80.1 | 78.7 | 73.3 |
| (vi) VideoGen-Agent-RL-Toolset 1 — w/o tool reward | 74.3 | 70.2 | 60.2 | 60.9 | 82.7 | 78.8 | 71.2 |
| (vii) VideoGen-Agent-RL — single-task variant | 85.5 | 74.5 | 65.9 | 67.3 | 84.4 | 80.3 | 76.3 |
| (viii) VideoGen-Agent-RL-Toolset 1 (Full, multitask) | 80.4 | 75.1 | 65.3 | 67.9 | 83.1 | 81.7 | 75.6 |
Prompt rewriting alone (ii) adds only +0.8 and zero-shot tool use (iii) adds only +3.0 — both fall well short of the full agent's +19.1, confirming that the gain comes from learned tool use rather than prompt elaboration or tool access alone. Removing either reward component (v, vi) drops the overall score, with the tool reward carrying the larger contribution. The single-task variant (vii) edges out the multitask agent (viii) by +0.7 overall, but the shared multitask policy supports all six categories with one agent at roughly the same overall quality.
Sample videos across categories
Six prompts, one per task category. Left: Seedance 1 Pro Fast single-pass baseline. Middle: Seedance 2 Pro Fast single-pass baseline (same prompt, stronger backend, still one call). Right: VideoGen-Agent (ours) — the final clip from the agent trajectory.
An elderly Tai Chi practitioner in a white tunic performs 'White Crane Spreads Its Wings' in a quiet courtyard.
Vigil (from the game Arknights) draws his crossbow in a fluid motion, cloak billowing as he fires a bolt trailing dark energy, his single visible eye narrowing while architecture crumbles behind him.
On a sunny Paris rooftop terrace at golden hour, a young actor performs in front of the camera as Mati Diop adjusts the cinema camera, Ladj Ly crouches beside the video-village monitor pointing at the playback, and Audrey Diwan leans over his shoulder pointing at the same screen to discuss the take.
Glass cracking, a cricket ball breaks a pavilion window and the scorer inside shields his ledger.
In a 12-second video, 0-4s: a heavy rain gutter clogged with leaves and overflowing. 4-8s: the weight cracks the bracket. 8-12s: the gutter section swings down and dumps water and leaves on the walkway.
A 0.3 kg rubber ball at 8 m/s hits the edge of a 3 kg metal cube and the cube starts rotating. Please maintain physics realism.
Extensibility — adding a new tool family to the agent
A test of the agent framework's extensibility: mid-training we introduce a new action-to-video tool family the agent has never seen before. Rather than touch its architecture or its reward, we just expose two new tools — pi0.5 (predicts a 60-step joint trajectory from an image + instruction) and ctrl-world (an action-conditioned world model that renders that trajectory as video) — and continue training on a handful of robotic-manipulation examples. The trained agent learns to route embodied-manipulation prompts through this new pipeline instead of the standard T2V/I2V route, without any change to how it reasons over the other six categories. This mirrors the paper's Extensibility and Generalization to Unseen Task Combinations section: the same policy absorbs a new tool family and produces videos grounded in a real motor plan, not just a first-frame image guess.
Four manipulation prompts routed through the newly added action-to-video tool chain. Columns left → right: the natural-language prompt, the I2V input frame (shared starting image), Ground truth (real robot recording), VideoGen-Agent with the new tool family (pi0.5 → ctrl-world), and Wan 2.5 I2V as a first-frame-only baseline for comparison — re-run with the subject explicitly pinned to "a robotic arm gripper" so the comparison isolates the missing motor plan rather than a vague noun.
| Prompt | I2V input | Ground truth | VideoGen-Agent (ours) | Wan 2.5 I2V |
|---|---|---|---|---|
| Pick & Placepick up the blue block and place in white plate |  |
|||
| Pick & Placepick up the red block and place in white plate |  |
|||
| Towel Foldfold the towel |  |
|||
| Wipe Tablemoving the towel from right to left |  |
Ground-truth recordings are 320×192 at 4 fps (raw robot camera); the agentic pipeline inherits that resolution since ctrl-world generates in the same space as its training data. The Wan 2.5 I2V baseline runs at 1238×744, 24 fps, upscaled from the same 800×480 first-frame input. All cells displayed at uniform width — black bars are letterboxing, not crop.