Building an AI Video Agent: How to Generate MP4s from LLM Prompts with VideoJSON

Giving an AI agent the ability to "see" is common, but giving it the ability to create high-quality video has traditionally been a nightmare of shell scripts and proprietary APIs. If you've ever tried to make an LLM emit a valid FFmpeg command, you know the pain: one misplaced flag and the whole pipeline collapses. The syntax is brittle, the feedback loop is slow, and the lack of structured data makes it impossible for an agent to "reason" about the timeline it's building.

To build a truly robust AI Video Agent, you need a medium that is as structured as it is expressive. You need a way to treat video as data, not as a side effect.

In this guide, we'll look at why VideoJSON is the perfect interface for LLM-driven video generation and how you can build a pipeline that turns a simple text prompt into a rendered MP4 in seconds.

The Problem with LLMs and FFmpeg

LLMs are excellent at generating structured text (JSON, Markdown, Code) but struggle with the "stringly-typed" nature of traditional video tools. When you ask an agent to generate a video via FFmpeg, you're asking it to manage complex filter graphs, timestamp offsets, and codec flags in a single, massive command string.

This approach is brittle for several reasons:

1. No Validation: You can't easily validate a 500-character FFmpeg string before running it. If the agent halluncinates a flag, the render fails with a cryptic error.
2. State Management: Keeping track of which layer is at which timestamp across multiple commands is impossible for an agent's context window. FFmpeg doesn't have a concept of a "project state."
3. Licensing & Portability: Running FFmpeg on a server is heavy and expensive. Furthermore, you certainly can't run it in a user's browser, which limits your ability to offer low-latency previews.

Compared to something like Remotion, which requires an entire React runtime to define a scene, VideoFlow uses a portable JSON schema. This makes it a Remotion alternative that is uniquely suited for AI agents: the agent emits JSON, and the renderer handles the rest. The agent doesn't need to learn a framework; it just needs to learn a schema.

Why VideoJSON is the Perfect Agent Interface

VideoFlow's core philosophy is that a video is a document. By using the @videoflow/core builder, we can create a "tool" for our agent that maps directly to cinematic primitives. Instead of a black box, the agent sees a clean, typed API. It doesn't need to know how to encode an H.264 stream; it just needs to know how to call $.addText or $.addVideo.

The Agent Tool Definition

Imagine giving your agent a tool called create_video_scene. The schema for that tool maps directly to our Builder API. Here is how you might wrap VideoFlow for an agent using the functional builder pattern:

import VideoFlow from '@videoflow/core';

async function renderAgentScene(agentData) {
  const $ = new VideoFlow({ width: 1080, height: 1920, fps: 30 });

  // Add a background image from a generated URL
  const bg = $.addImage(
    { fit: 'cover', opacity: 0.8 },
    { source: agentData.backgroundImageUrl }
  );

  // Add AI-generated captions with word-level timing
  $.addCaptions(
    { 
      fontSize: 6, 
      color: '#FF5A1F', 
      position: [0.5, 0.8], 
      fontWeight: 700 
    },
    { captions: agentData.transcript }
  );

  // Apply a cinematic transition to a headline
  const title = $.addText({ text: agentData.headline, fontSize: 8 }, {
    transitionIn: { transition: 'blurResolve', duration: '800ms' }
  });

  // Advance the flow
  $.wait(agentData.totalDuration);
  
  // Compile to portable VideoJSON
  return await $.compile();
}

This code is readable, testable, and—most importantly—easy for an LLM to generate. Because the builder handles the frame-math, the agent only has to focus on the content and the timing.

Rendering the Output: The Three-Renderer Rule

One of the most powerful features of VideoFlow is its three official renderers. Because the agent's output is a portable JSON file, you can choose where the rendering happens based on your cost and UX requirements.

1. Server-side Rendering: Use @videoflow/renderer-server in a Lambda or Node.js environment to batch-process videos. This is perfect for "Social Media Factory" use cases where you generate hundreds of personalized ads per hour.
2. Browser-side Export: Use @videoflow/renderer-browser to let users export the video directly in their tab using WebCodecs. This reduces your server costs to zero and provides an instant gratification loop for the user.
3. Live Preview: Use @videoflow/renderer-dom to show the agent's progress in real-time at 60fps while it's still "thinking." The user can watch the video being built, layer by layer.

As we discussed in our guide on why AI agents should speak VideoJSON, this separation of concerns is what allows for truly scalable video automation. You can debug the JSON in the Playground before ever spending a cent on GPU compute.

How VideoFlow Handles the Heavy Lifting

When your AI Video Agent emits a VideoJSON document, VideoFlow takes care of the complex math that usually breaks automation pipelines:

• Resolution Independence: The agent uses em units (1% of width). If the agent builds a 1080p video, you can render it at 4K without changing a single line of code. This ensures your automated content looks sharp on every screen.
• Preset Library: With 27 transition presets (like blurResolve, glitchResolve, and lightSweepReveal) and 42 GLSL effects, the agent can create cinematic content without knowing a lick of shader code or GLSL.
• Frame-Perfect Timing: The $.wait() and $.parallel() primitives allow the agent to reason about time linearly, rather than calculating absolute frame numbers. It's like writing a script instead of a spreadsheet.

Build Your Own Agent

Building a video-capable agent is no longer a multi-month engineering project. By combining an LLM with the VideoFlow toolkit, you can turn text into high-fidelity video with the same ease that you generate a blog post or a tweet. The open-source nature of the core package means you have full control over your pipeline, with no vendor lock-in and no proprietary black boxes.

Ready to start building?

• Explore the Playground to see how VideoJSON looks in action.
• Check out our Getting Started guide to install the core package.
• Star the project on GitHub to follow our open-source journey.

By treating video as data, we're moving past the era of fragile shell scripts and into the era of programmatic, agent-driven creativity. The future of video isn't just recorded; it's computed.