Modular Video Generation: Using Groups to Build Complex AI-Driven Timelines

Generating a simple 5-second video with an AI agent is relatively straightforward. But as soon as you ask an LLM to build a multi-scene explainer video or a data-heavy recap, the token complexity explodes. The agent starts losing track of frame offsets, layer indices, and timing logic.

To solve this, we need to move away from flat timelines and embrace modular video generation. By using VideoFlow Groups, you can allow your AI agent to build videos hierarchically—composing isolated "scenes" that are then layered and timed as single units.

The Complexity Wall in AI Video Pipelines

Most video-as-code tools require you to specify the startTime and duration for every single layer relative to the global zero-second mark. For an AI agent, this is a massive cognitive load. If it adds a 3-second intro, it must then remember to offset every subsequent layer by exactly 3 seconds. One math error and your entire ai video pipeline falls out of sync.

VideoFlow's $.group() API changes this by creating a private coordinate space and timing flow. Inside a group, the clock starts at zero. The agent can focus on building one scene at a time without worrying about the global timeline.

Scene-Based Composition with GroupLayer

When you use $.group(), you are creating a GroupLayer. This layer acts as a container for other layers (text, images, video, etc.). The beauty of this approach is that the group itself can have its own position, opacity, and even transitions.

If your agent defines a "Title Scene" inside a group, it can then apply a blurResolve transition to the entire group. Every layer inside that group will transition in unison. This is far more efficient than asking an LLM to apply identical transitions to six different layers manually.

Example: A Modular Scene Builder

Here is how an agent can use the VideoFlow builder API to create a modular scene:

import VideoFlow from '@videoflow/core';

const $ = new VideoFlow({ width: 1920, height: 1080 });

// Scene 1: The Intro
$.group({ opacity: 1 }, { transitionIn: { transition: 'fade' } }, ($scene) => {
  $scene.addText({ text: 'The Future of Video', fontSize: 8, position: [0.5, 0.4] });
  $scene.addText({ text: 'Powered by AI', fontSize: 4, position: [0.5, 0.6] });
  $scene.wait('3s');
});

// The global flow has now advanced by 3 seconds automatically.

// Scene 2: Data Visualization
$.group({ scale: 0.9 }, {}, ($scene) => {
  $scene.addShape({ fill: '#FF5A1F' }, { shapeType: 'rectangle' });
  $scene.wait('5s');
});

Why VideoJSON is the Perfect AI Output

Because VideoFlow compiles to a portable VideoJSON document, your agent doesn't need to be a React expert. It just needs to output a structured JSON schema. This makes it trivial to integrate with LLM tool-calling.

By treating modular video generation as a hierarchical data problem, you gain several advantages:

1. Parallel Generation: You can have multiple agent calls generating different scenes (VideoJSON fragments) and then merge them into a single master project.
2. Diffable Timelines: Since the output is JSON, you can easily diff two versions of a video to see exactly what changed in the timing or properties.
3. Renderer Portability: The same scene can be rendered on the server via @videoflow/renderer-server or previewed in real-time using the Playground.

Scaling Your Video Infrastructure

As you move from simple scripts to full-scale automation, the need for a robust rendering engine becomes clear. VideoFlow allows you to render MP4s in Node without FFmpeg, leveraging headless Chromium for pixel-perfect accuracy.

This modular approach is the key to building "Video Agents" that can iterate on content. Instead of re-generating a 60-second video from scratch, the agent can simply swap out one group in the JSON and trigger a re-render.

Get Started with Modular Video

Ready to build your own AI video pipeline? The best way to start is by exploring our documentation and experimenting with groups in the interactive playground.

If you're building a commercial product and need a full-featured timeline interface for your users, check out the React Video Editor. For everything else, the core engine is open-source and ready for your most ambitious agentic workflows.

Check out the source and join the community on GitHub.