VideoGen-Agent:
Reinforcing Video Generation Agents

Method

VideoGen-Agent formulates video generation as a multi-turn tool-use problem. Given a prompt, the agent sequentially decides whether to seek additional context, generate a video candidate, or verify and refine an intermediate result. Tools are organised into three functional groups:

Training

Starting from Qwen3-VL-8B-Instruct, we train on 16K teacher trajectories from Gemini 3.1 Pro and Claude Opus 4.7, then apply multitask GRPO on a separate set of 8K prompts. The 24K prompts span six task categories. The hybrid reward combines tool-call validity, category-specific video quality, and task-appropriate tool use, with weights 0.1, 0.5, and 0.4. Group-relative advantage normalization is followed by task-level normalization.

SFT uses two epochs with a learning rate of 5 × 10⁻⁵; RL uses one epoch at 10⁻⁶ and six usable rollouts per prompt. Failed action spans are masked during SFT. During RL, trajectories that fail because of external tool errors are resampled before group statistics are computed.

Per-category results on VABench

Scores are the Gemini 3.1 Pro judge mean in [0, 100], averaged within each category. Higher is better. Overall is the unweighted mean across all six categories. Bold marks the best score and underline marks the second best.

Toolset 1 is the training-time stack VideoGen-Agent was optimised against. Toolset 2 swaps in stronger generation tools at inference time without any additional agent training, demonstrating that the trained agent benefits from subsequent advances in generation tools. Hover the small ? at the top-right of each Toolset name for the exact models (with links).

Ablation study

Rows (i)–(iv) and (viii) compare vanilla T2V, prompt rewriting, a fixed agentic workflow, SFT, and RL under the Toolset 1 configuration. Rows (v) and (vi) remove individual reward components; row (vii) reports the single-task variant. Overall is the unweighted mean across all six categories. The highest score in each column is bolded.

The fixed-workflow baseline uses the same Qwen3-VL-8B-Instruct backbone and Toolset 1 without task-specific training. For MI, the backbone receives the required entity count N, lists N names, retrieves one reference image per entity, and calls video_gen_multiple_reference for reference-to-video generation. For CS, ReAct is capped at three iterations; other tasks execute their prescribed workflow once. The paper appendix details these task-specific budgets and the single-task training setup.

Prompt rewriting improves the overall score from 56.5 to 57.3, while the fixed workflow reaches 64.1. SFT further improves performance to 69.2, and RL adds 6.4 points to reach 75.6, with gains across all six categories. The advantage over the fixed workflow supports the usefulness of training under these task-specific baseline budgets.

Removing the VLM reward reduces the overall score to 73.3, while removing the tool reward lowers it to 71.2; both ablations degrade all six categories. Single-task agents undergo SFT and then RL using only their category-specific subsets, with other settings unchanged. They achieve 76.3 overall, 0.7 points above the shared multitask agent. Single-task training performs better on PK, MI, and CS, while multitask training leads on SI, PS, and MS and supports all six tasks with a single policy at comparable overall performance.

Generalization to an unseen task combination

Procedural Identity (PI) combines procedural knowledge with single-entity identity. We construct 100 held-out prompts from 20 specialized actions or processes and 20 named entities, disjoint from SFT, RL, and VABench prompts. No training prompt combines these requirements. The system prompt provides general composition guidance, but no explicit workflow for this combination.

Without additional training on PI, the RL agent improves over SFT and scores close to the explicitly composed fixed workflow. The fixed baseline executes Search-Text, Search-Image, and reference-conditioned generation once with Toolset 1; its generation-call count is not matched to the RL trajectory. Invocation rates record whether a tool was called at least once, not whether both task requirements were satisfied.

Human evaluation and judge validation

Human preference: we conduct two side-by-side comparisons under the same protocol, each using 100 randomly sampled VABench video pairs and four raters. Each rater selects one preferred video per pair, with no tie option, yielding 400 judgments per comparison. VideoGen-Agent-Toolset 2 is preferred over Seedance 2.0 in 84.3% of judgments, while VideoGen-Agent-Toolset 1 is preferred over its standalone text-to-video generator, Seedance 1.0 Pro Fast, in 88.0% of judgments.

Judge–human agreement: a separate study uses 100 pairs per category (600 total), balanced across the strongest standalone baseline, SFT, and RL. Three blinded raters independently assess each pair, and Gemini 3.1 Pro agrees with their majority preference on 92.0% of pairs. Equal VLM scores trigger a separate forced-choice comparison under the same rubric; these choices are used for agreement, not to replace benchmark scalar scores.

Agreement is lowest on procedural knowledge (PK), which remains a challenge for the judge. Agreement on the other five categories ranges from 88.0% to 100.0%.

Robotic manipulation with an additional tool family

Starting from the RL-trained checkpoint, we fine-tune the policy on 1,000 additional robotic-manipulation examples. Given an input image and instruction, the agent calls π_0.5 to predict a joint-action trajectory and passes the actions to Ctrl-World to generate a video. These qualitative examples show that the fine-tuned policy can invoke the two tools in sequence. This robotics checkpoint is separate from the checkpoint evaluated on the six video-generation tasks.

The four examples cover placing a blue or red block in a white plate, folding a towel, and moving a towel from right to left. Columns show the prompt, input image, ground-truth recording, VideoGen-Agent output, and Wan 2.5 I2V output. The I2V baseline uses the same initial image and explicitly names a robotic arm gripper as the acting subject.

Ground-truth recordings and agent-generated clips are 320×192 at 4 fps. Wan 2.5 I2V clips are 1238×744 at 24 fps, using an 800×480 input derived from the same initial frame. Ground-truth clips last 20.5–29.5 seconds, agent clips 12–15 seconds, and baseline clips approximately 5 seconds; playback is not temporally aligned. Full fields of view are preserved and resized only for display. These examples do not constitute a controlled quantitative evaluation of robotic task success or physical robot execution.