Generating a video automatically sounds straightforward at first: take some text, add a voice-over, place a few images on screen, and export the result.

In practice, producing a coherent video requires coordinating several different systems. The narration has to be timed correctly, each section needs a relevant visual, stock footage must be searched and evaluated, generated images need usable prompts, media must be fitted to the video format, and everything has to remain synchronised throughout the final render.

I built an automated video-generation tool to explore this problem.

The system takes a written script and transforms it into a narrated video using a combination of AI-generated images, stock photography, stock video, text-to-speech, and automated editing.

The most important part of the project is not any individual AI model. It is the pipeline that turns several unpredictable media-generation processes into one structured and repeatable workflow.

The Overall Pipeline

At a high level, the system follows this sequence:

Written script
      ↓
Voice-over generation
      ↓
Narration timing analysis
      ↓
Visual scene planning
      ↓
Media search and generation
      ↓
Asset processing
      ↓
Timeline construction
      ↓
Video rendering

Each stage produces structured information that can be used by the next.

This separation matters because each part of the process solves a different problem. Text-to-speech creates audio, but it does not know which image should appear at a particular moment. A stock-media search API can return relevant videos, but it does not know how long each clip should remain on screen.

The orchestration layer connects these individual capabilities.

Starting With the Script

The script is the source of truth for the entire video.

It defines the subject, the narration order, the tone, and the concepts that need to be represented visually. However, a normal script does not contain enough information to construct a video timeline.

Consider a narration such as:

The first commercial computers filled entire rooms. Today, far more processing power fits inside a mobile phone.

A person editing the video might naturally divide that into several visual ideas:

• An early room-sized computer.
• Engineers operating old computing equipment.
• A transition to a modern smartphone.
• A close-up of a mobile processor.

The original text does not explicitly contain those editing decisions. The pipeline therefore needs to convert the narration into a more detailed visual plan.

Generating the Voice-Over First

One of the key architectural decisions was to generate the voice-over before finalising the visual timeline.

The duration of written text is difficult to predict accurately. Two paragraphs with similar word counts may have different speaking times because of punctuation, sentence structure, pauses, abbreviations, and the selected voice.

By generating the narration first, the system can work with the real audio duration rather than estimating it from the script.

The voice-over becomes the timing foundation for the rest of the video.

Once generated, the system can analyse or derive information such as:

• Total narration duration.
• Sentence or phrase boundaries.
• Approximate start and end times.
• Natural pauses.
• Sections that require longer visual coverage.
• Sections where faster media changes may be appropriate.

This reverses the usual temptation to select visuals immediately. Instead, the media timeline is constructed around the finished narration.

Using AI to Assign Timestamps

The next stage converts the script and narration timing into a set of timed scenes.

The AI is not simply asked to summarise the script. It is asked to act more like an editor by identifying where one visual idea should end and another should begin.

A simplified output might look like this:

{
  "scenes": [
    {
      "start": 0.0,
      "end": 5.8,
      "narration": "The first commercial computers filled entire rooms.",
      "visualConcept": "Large early computer in a 1950s computing laboratory"
    },
    {
      "start": 5.8,
      "end": 11.4,
      "narration": "Today, far more processing power fits inside a mobile phone.",
      "visualConcept": "Modern smartphone followed by a close-up of a processor"
    }
  ]
}

The important feature is that the output is structured.

The system needs to know more than what each scene is about. It also needs a usable start time, end time, visual concept, preferred media type, and enough context to search for or generate that media.

This scene plan becomes the central contract between the language model and the rendering system.

Separating Narration From Visual Meaning

A script sentence is not always a good stock-media search query.

For example, a line such as:

The market was beginning to lose confidence.

Searching for that full sentence would probably produce poor results. The system instead needs to extract a visual interpretation, such as:

Falling financial chart
Concerned investors
Empty trading floor
Newspaper headlines about economic uncertainty

The AI therefore acts as a translation layer between narrative meaning and searchable visual concepts.

This is one of the most useful roles for a language model in the pipeline. It can take abstract narration and produce concrete descriptions that are more suitable for stock-media search or image generation.

The model can also classify each scene according to the type of visual that is likely to work best.

For example:

{
  "visualType": "stock_video",
  "searchQuery": "busy modern data centre server racks",
  "fallbackImagePrompt": "cinematic view inside a modern data centre"
}

That allows the pipeline to choose different media strategies without hard-coding every decision.

Choosing Between Stock Video, Stock Images, and AI Images

Not every scene should use the same type of media.

Stock video is useful when motion adds value. Examples include cities, transport, machinery, nature, crowds, workplaces, or general establishing shots.

Stock photography may be preferable when a strong still image exists and movement is not essential.

AI-generated imagery is particularly useful when the required visual is too specific, abstract, historical, fictional, or difficult to find in a stock library.

The pipeline can therefore treat media selection as a prioritised decision.

A scene may follow a sequence such as:

Search for suitable stock video
        ↓
Search for suitable stock image
        ↓
Generate an AI image
        ↓
Use a controlled fallback visual

The exact ordering can vary depending on the project, but the underlying principle is that the system should have more than one way to satisfy a visual request.

This makes the pipeline more resilient. A missing stock result does not have to stop the entire video.

Searching Pexels Programmatically

For stock media, the system uses Pexels as one of its external media sources.

Each scene contains a search phrase derived from the narration. The application sends that query to the stock-media service and receives a set of candidate images or videos.

However, the first returned result is not necessarily the best result.

The system needs to consider properties such as:

• Media orientation.
• Resolution.
• Duration.
• Aspect ratio.
• Relevance to the scene.
• Whether the clip contains enough usable footage.
• Whether it can be cropped without losing the subject.
• Whether the visual style fits surrounding scenes.

For video results, duration is especially important. A four-second stock clip may not be suitable for a scene that lasts twelve seconds unless it can be looped, slowed, extended, or combined with another asset.

The pipeline therefore treats the search result as a candidate rather than a final editorial decision.

Creating AI Image Prompts

When a scene requires generated imagery, the visual concept must be converted into a stronger image prompt.

A useful image prompt typically needs more detail than the original narration provides.

The pipeline may enrich the concept with information such as:

• Subject.
• Environment.
• Time period.
• Camera angle.
• Lighting.
• Composition.
• Intended aspect ratio.
• Level of realism.
• Visual tone.
• Elements that should not appear.

For example, a simple concept such as:

An early computer

might become a more production-oriented instruction describing a large room-sized machine, rows of control panels, period-appropriate operators, documentary-style composition, and space for cropping.

This prompt generation stage is kept separate from scene planning. The scene planner decides what needs to be shown, while the image-prompt stage decides how to express that concept to an image model.

That separation makes both stages easier to improve.

Asset Normalisation

The media returned by different sources is rarely consistent.

A stock image may be landscape, while an AI image may be square. A stock video may be vertical. Some clips may have unusual frame rates, dimensions, or compression formats.

Before media can be placed into the final timeline, it needs to be normalised.

This stage can include:

• Resizing.
• Cropping.
• Padding.
• Converting formats.
• Adjusting frame rates.
• Trimming video duration.
• Removing source audio.
• Creating motion from still images.
• Ensuring consistent resolution.
• Preparing assets for the renderer.

Still images can be made more visually engaging through controlled movement such as slow zooming or panning. This prevents the final video from appearing like a static slideshow while still allowing images to cover longer narration sections.

The normalisation layer also protects the renderer from having to handle every possible input format directly.

Constructing the Timeline

Once the system has the voice-over, scene timestamps, and processed media, it can construct the video timeline.

Each scene becomes a timeline segment containing:

{
  "start": 18.2,
  "duration": 6.4,
  "media": "scene_04.mp4",
  "transition": "crossfade",
  "sourceType": "pexels_video"
}

The timeline builder is responsible for ensuring that:

• Scenes appear at the correct time.
• There are no unintended gaps.
• Visuals do not overlap incorrectly.
• Media fills the required duration.
• Transitions do not alter the overall synchronisation.
• The final visual track matches the narration length.

This is where small timing errors can become noticeable.

If every scene is only a fraction of a second too long, the final visual track can drift significantly by the end of the video. The system therefore needs to treat timestamps as part of a continuous timeline rather than a collection of approximate durations.

Handling Media That Is Too Short

One recurring challenge is that the selected media may not match the duration of the scene.

If a stock video is shorter than its assigned segment, the pipeline needs a strategy.

Possible processing decisions include:

• Trimming a longer clip.
• Slowing a clip slightly.
• Looping footage when the loop is not visually obvious.
• Combining multiple related clips.
• Extending a still image using camera motion.
• Replacing the asset with a better candidate.

The appropriate decision depends on the content.

A looping ocean clip may look natural. A looping clip of a person turning their head may be immediately noticeable. This means that duration correction cannot always be handled with one universal rule.

The media type and subject need to influence the processing strategy.

Generating the Final Composition

The rendering stage combines all prepared elements into a final composition.

The main layers are generally:

Visual media
Narration audio
Optional background music
Text or captions
Transitions
Branding or overlays

The narration remains the primary timing track. Other elements are aligned to it.

The renderer creates the frames, applies any transitions or motion, mixes the audio, and exports the final file.

At this stage, the artificial intelligence is no longer making creative decisions. The renderer works from the structured timeline created earlier in the pipeline.

This distinction is important.

The AI proposes the editorial structure, but deterministic software is responsible for producing the actual video.

Why Structured AI Output Matters

Without structured output, the AI might return a useful description of the video but leave the application unable to act on it.

A response such as:

Show an old computer for a few seconds, then transition to a smartphone.

is understandable to a person, but it is not precise enough for an automated renderer.

A structured response can include:

{
  "start": 0,
  "end": 5.8,
  "mediaType": "stock_image",
  "searchQuery": "1950s room sized computer",
  "transitionOut": "crossfade"
}

The language model is therefore used to produce data rather than final execution instructions.

The application validates that data and translates it into its own internal timeline representation.

This is similar to working with any external service: the response must satisfy an expected contract before the application can trust it.

Validation and Failure Recovery

AI responses and media APIs both fail in unpredictable ways.

The scene planner might produce overlapping timestamps. A stock-media search may return no useful results. An image-generation request may fail. A downloaded video may be corrupt or have an unsupported codec.

The pipeline needs to expect these problems.

Validation can check whether:

• Every scene has a valid start and end time.
• Scenes remain within the narration duration.
• The timeline contains no accidental gaps or overlaps.
• Every visual concept has a search query or image prompt.
• Downloaded assets exist and are readable.
• Media dimensions meet minimum requirements.
• Video clips have valid durations.
• Generated images have completed successfully.
• Every scene has a usable final asset.

A failed scene should ideally be retried or replaced without restarting the entire project.

That means the generation state has to track progress at the scene level.

Maintaining a Project State

The pipeline stores information about each stage of the video-generation process.

A project state might contain:

• Original script.
• Voice-over file.
• Narration duration.
• Timed transcript.
• Scene definitions.
• Search queries.
• Candidate media.
• Selected media.
• Generated-image prompts.
• Processing status.
• Render status.
• Errors and retry counts.

This makes the process observable.

Instead of treating the entire video as one opaque generation request, the application can show which scenes are complete, which are still searching for media, and which need to be regenerated.

It also allows work to resume after a failure without repeating expensive completed stages.

Reproducibility and Caching

Several parts of the pipeline involve remote services and potentially costly operations.

Generating the same voice-over, downloading the same stock video, or recreating the same AI image unnecessarily would make development slower and increase usage costs.

Caching therefore becomes an important architectural feature.

The system can reuse outputs based on identifiers derived from inputs such as:

• Script text.
• Voice settings.
• Search query.
• Selected media URL.
• Image prompt.
• Resolution.
• Render settings.

This is especially useful during development. A change to one scene should not require every voice, image, and video asset to be generated again.

The project becomes a dependency graph rather than a single all-or-nothing process.

AI as an Editor, Not the Renderer

One of the clearest lessons from the project is that AI works best here as an editorial and semantic layer.

It can:

• Interpret the script.
• Identify visual changes.
• Create scene descriptions.
• Produce search queries.
• Write image-generation prompts.
• Decide which media type is appropriate.
• Help repair malformed planning output.

It should not be responsible for the precise mechanics of video rendering.

Frame timing, media conversion, file handling, audio mixing, and export are better handled by deterministic software.

This creates a useful division of responsibilities:

AI:
Understands meaning and proposes creative decisions

Application:
Validates decisions and manages the workflow

Renderer:
Produces the final deterministic output

This separation makes the system more reliable and easier to debug.

Avoiding a Single Giant Prompt

It would be possible to send the entire script to a model and ask it to return every scene, timestamp, search query, image prompt, and editing decision in one response.

For short videos, that may sometimes work.

For larger projects, it becomes increasingly fragile.

A more dependable architecture divides the work into focused stages:

1. Analyse the script.
2. Align the narration with timing information.
3. Divide the content into scenes.
4. Develop visual concepts.
5. Generate media-specific instructions.
6. Validate the results.
7. Build the timeline.

Each stage has a smaller task and a more limited output format.

This also allows individual parts of the system to evolve independently. The timestamp planner can be improved without replacing the stock-media search logic. The image-prompt generator can be changed without affecting audio generation.

Keeping Human Review Possible

Although the pipeline can operate automatically, the intermediate scene plan is also useful as an editing interface.

Before rendering, a user could review:

• Scene boundaries.
• Visual descriptions.
• Stock-media selections.
• Generated images.
• Search terms.
• Scene durations.
• Transitions.

This provides a point where automation and manual control can meet.

The system can generate a complete first pass, while still allowing a person to replace an unsuitable image or change the visual concept for a specific scene.

This is often more practical than expecting every automatically selected asset to be perfect.

The Broader Architecture

The pipeline follows a pattern that applies to many generative media systems:

Unstructured creative input
          ↓
Semantic interpretation
          ↓
Structured production plan
          ↓
External asset generation and retrieval
          ↓
Validation and normalisation
          ↓
Deterministic composition
          ↓
Final media output

The AI does not generate the entire finished product directly.

Instead, it produces structured decisions that traditional software can inspect and execute.

This architecture can be used for more than narrated videos. Similar systems could generate presentations, advertisements, product demonstrations, educational content, social-media clips, or visual summaries.

Final Thoughts

The challenge of automated video generation is not simply producing images or converting text to speech.

The difficult part is synchronisation.

The system has to connect the meaning of a script with the timing of a voice-over, the availability of stock media, the capabilities of image-generation models, and the strict requirements of a video timeline.

By separating the process into planning, timing, media selection, asset processing, and deterministic rendering, the application becomes much more manageable.

The language model provides interpretation and editorial decisions. Stock-media services and image models provide raw visual assets. Text-to-speech provides narration. The pipeline coordinates them all into a single timeline.

That orchestration layer is what turns a collection of AI services into an actual video-production system.