48 video ads from 24 minutes of footage: the Remotion pipeline
Remotion renders video from React, which makes it a good fit for ad work where you need the same thing forty times with one variable changed. Here's the whole pipeline, including the parts that aren't Remotion at all.
To generate video ads with Remotion you write the ad as a React component, drive every animation from useCurrentFrame(), and render the same composition into each aspect ratio you need. That part is straightforward. The work that decides whether the ads are usable happens around it: cutting raw footage at points where the speaker has actually finished a thought, getting the loudness and the room out of the audio, and burning in captions that are spelled correctly and land on the right frame.
We built 48 ads out of 24 minutes of phone footage this way, plus 20 more from an earlier shoot. Most of what follows is what went wrong the first time, because that turned out to be the useful part.
The short version
- Remotion is maybe a third of the pipeline. It renders the ad. Deciding where to cut, fixing the audio and checking the output are the other two thirds, and they are where the quality lives.
- Cut on the voice, not on the transcript. A transcript knows where a sentence ends as text. It cannot tell you whether the speaker had finished, and cutting a beat early is the thing that makes an ad sound amateur.
- Never normalise with single-pass
loudnorm. It is a dynamic filter: it returned clips 360ms short and compressed away the level fall that makes an ending sound final. - Fix caption text at the sentence level, not the token level. Whisper splits words unpredictably, so
"Adside.ai"can arrive as four tokens and a token-level rule silently never fires. - Measure the artifact, never the intent. Every expensive mistake came from trusting what a step meant to do rather than checking what it produced.
The file to hand your agent
Most people reading this will implement it with a coding agent rather than by hand, so the condensed version is a single markdown file with no marketing in it:
Paste that URL into Claude Code, Cursor or whatever you use, or download it and drop it in the repo. It carries the pipeline, the specific numbers, and the traps, in a form an agent can follow. The rest of this article is the same material with the reasoning attached, which is worth reading if you are going to be the one debugging it.
When code beats an editor
Generating video with code is worth it when you need many variations of a structured video: one ad in three aspect ratios, twelve hooks over one body, forty cuts of a single interview. The win is repeatability. Change the end card once and 48 ads update.
It is not worth it for a one-off brand film. An editor will do that better and faster, and nothing about the approach helps you.
The volume case matters more than it used to because creative volume is now the main lever on Meta: you need enough distinct concepts in rotation to keep finding winners, which is a production problem before it is a media-buying one. We've written separately about how many creatives you actually need and how to structure the tests.
One thing to settle before you build anything: Remotion is free for individuals, non-profits and for-profit companies with up to 3 employees. For-profit companies with 4 or more employees need a paid Company License.¹ Worth knowing on day one rather than the day the pipeline becomes load-bearing.
Remotion for ad work
A composition is a React component plus a frame rate, dimensions and a duration. useCurrentFrame() tells the component which frame it is drawing, and the renderer walks the frames and encodes them.²
The structural decision that matters is keeping content out of components. An ad should be a data file (which clip, which copy, which format); the template is code. When those mix, ad number thirty is a copy-paste of ad twenty-nine and nothing can be changed globally.
Four rules that are specific to Remotion and cost us time:
- CSS transitions and animations do not render. The renderer jumps between frames, so anything time-based in CSS produces nothing at all. Every animation has to be driven from the frame number.
<Video>takesobjectFitas a prop, not instyle. Set it instyleand your footage letterboxes silently.- Round computed pixel values. A sub-pixel
gapcan serialise into exponential notation, which the style parser drops. Ours became zero and the words touched. - Use
<Video>and<Audio>from@remotion/media. Plain HTML tags do not sync to the frame timeline.
Cut on the voice, not the transcript
This is the part that decides whether the ads feel professional, and it is the part everyone gets wrong first, including us.
A transcript tells you where a sentence ends as text. It cannot tell you whether the speaker had finished. English statements close on a terminal fall: pitch declines across the last syllables, the level tails off, and a pause follows. Cut before that lands and the ad sounds interrupted even though every word is present.
So measure three things at any candidate cut point, from the audio itself:
| Signal | What it tells you | Threshold we use |
|---|---|---|
| Pause after the cut | Whether he stopped at all | ≥0.3s, and ≥0.7s is conclusive on its own |
| Pitch fall | Whether the sentence closed or is still running | Closing F0 ≤93% of the utterance's own median |
| Level decay | Whether the delivery tailed off | ≥3dB drop into the cut |
Below a 0.7s pause, require a pause plus either the pitch fall or the decay. Two implementation traps cost us most of a day each:
If your probe point lands inside a speech region, do not measure the pause to the next region. That counts the rest of the sentence he is still saying as silence, and a mid-word cut scores as a 1.9-second pause. Return zero instead.
And autocorrelation pitch detection reads an octave low more often than you would expect. Everything measured a flat 82Hz until we added octave correction; the real voice was 110–140Hz falling to 83Hz at a terminal fall. Without that correction every ending looks equally flat, so the whole check silently does nothing.
Cadence proves the speaker stopped. It cannot prove the thought was complete.
That distinction matters. A half-second breath mid-sentence passes every acoustic test, which is how we shipped an ad ending "…paired with a human expert we managed." The fix is to also require the transcript to end on terminal punctuation, or at least not on a word that cannot end a clause.
Trim the start too. Leading silence is dead air at the top of the ad, and it also makes Whisper smear the first words backwards across it: 1.7 seconds of room tone made it place "I believe that" at 0.00s when the speaker doesn't start until 1.71s. Beginning the cut about 120ms before the first word fixes the caption timing and the dead air together.
Finally, accept that some takes cannot be saved. Three of our recordings had no point at which the voice landed, because the camera was stopped while he was still talking. That is a shooting problem, and no amount of editing fixes it. A beat of silence before you stop recording is the cheapest thing anyone can do to make footage editable.
Audio is where ads get thrown away
Two mistakes here, both of which we made.
Do not normalise with single-pass loudnorm. It is a dynamic filter, and two things go wrong that are invisible unless you measure the delivered file. Its lookahead is not flushed, so clips come out short: ours lost 360ms, which was the end of the last word, while the sidecar still recorded the duration that had been requested. Every downstream check therefore agreed the cut was clean. And being dynamic, it compresses, lifting the quiet tail of each sentence back toward speaking level and erasing the falling loudness that tells a listener a thought has finished.
Instead, measure integrated loudness with ebur128, compute a fixed gain, and apply it linearly. EBU R128 is the broadcast loudness standard the tooling implements;³ −16 LUFS is a reasonable target for social delivery.
Two gotchas around that. FFmpeg writes measurements to stderr, not stdout: capture only stdout and you silently get your fallback value, which in our case produced an "+83dB correction". And sample peak is not true peak: limiters clamp the former, delivery specs are written in the latter, and the reconstructed waveform overshoots between samples. Limiting to exactly −1.5 delivered −0.7 dBTP, so give the limiter about a decibel of extra headroom.
Reverb: the intuitive tools are the wrong ones
Our footage was shot outdoors and in a live room, and the echo was the most common note we got. The obvious moves are a spectral denoiser or a downward expander, and both are wrong. A denoiser assumes the unwanted part is stationary, but reverb is a delayed copy of the voice itself, so it removes the wrong thing and leaves watery artefacts. An expander only ducks the tail between words, leaving the room under every syllable.
There's a subtler failure worth naming. An expander steepens the level decay by construction, which is exactly what an RT60 estimate measures. It scored beautifully on our metric while doing almost nothing audible: one clip measured 0.25s under the expander and 0.70s without it. If your processing can flatter your measurement directly, the measurement is not evidence.
The right tool is real dereverberation. WPE predicts late reverberation from the signal's own recent past and subtracts it, so what is removed is by construction a delayed copy of the voice.⁴ That took RT60 from 0.70s to 0.30s with the transcript word-identical. Afterwards, keep it minimal: a small dip around 300Hz if the room is boxy, a little presence around 4kHz, gentle levelling.
Do not add a high-pass afterwards. A 2-pole IIR rings at its corner frequency, and that ringing is itself a decaying low-frequency tail: it put RT60 back to 0.35–0.40s, undoing part of the fix.
Take the dead air out
Around 17% of our raw runtime was not speech: pauses between sentences, all of which were sitting in the ads. Cutting every gap over 0.26s down to 0.1s made them noticeably punchier without touching a word. On one clip that was 20.5s down to 16.6s.
Removing a pause from a locked-off talking head leaves a jump cut, which reads as a dropped frame. Change the shot size across the cut instead: step the framing between wide and about 6% punched in, instantly rather than eased. That turns each removed pause into a visible edit.
Captions, spelled correctly
Roughly 85% of social video is watched with sound off, and captions are worth about a 12% lift in view time,⁵ so burned-in captions are not optional. Transcribe locally with a Whisper build⁶ and use a mid-size model for anything burned into a frame. The small models are where "paid ads" becomes "Paydads".
Two structural points matter more than the styling.
Corrections have to be durable. Whisper repeats the same mistakes on the same voice, and fixing the caption file by hand works exactly until the clip is re-cut, at which point it is re-transcribed and every edit is silently lost. That happened to us three times before we moved corrections into a file that is re-applied automatically after every transcription.
Fix at the text level, not the token level. This is the single most useful thing in this section. Whisper splits words across tokens unpredictably and differently every time:
A rule written as ["@","site"] never fires if your normaliser strips @ to an empty string, which ours did. Match against the joined sentence and map the match back onto whichever tokens cover it, and tokenisation stops mattering. Watch two details: a multi-word replacement fuses into paidads unless you re-introduce the separator, and a denylist must never match the correct spelling. Ours flagged "Adside" itself, which made every clean caption a violation.
On timing, the target is that a caption page appears within 0 to +220ms of the words. Slightly late is invisible; early means the viewer reads the line and then waits for it. Snap per token against the audio rather than per utterance, because the gaps between tokens are exactly what Whisper gets wrong: on one take the speaker says "We're", pauses 1.6 seconds, then "building", and Whisper had stretched "'re" across the whole pause so there was no gap in caption time at all.
Finally, size captions by measurement rather than authoring them. Measure the widest unbreakable word, cap the font size so it fits the column, count the wrapped lines and shrink until the block fits the height. One CSS trap: white-space: pre preserves the leading space but forbids line breaks, so if every break opportunity sits inside such a span, wrapping can never happen and the text overflows however carefully you computed the line count.
Checks that stop bad ads shipping
Every rule above exists because something shipped wrong. The only way they hold at volume is as checks that block the render, and the only way to trust a check is to deliberately break it once and confirm it fires.
| When | Check | Why it exists |
|---|---|---|
| Before render | Clip does not end while the voice is going | Measured on the file that will render, not on what the cut intended |
| Before render | Transcript ends on a finished thought | Cadence alone passes a mid-sentence breath |
| Before render | No caption more than 220ms early | Reading a line then waiting for it is very visible |
| Before render | No wrong brand spelling in any caption | Fixes run at transcription time and can be raced by a re-cut |
| After render | Loudness consistent across the set | Ads play back to back; a 15 LU step is worse than either level |
| After render | No render older than its footage | Re-cutting a clip does not re-render the ads that use it |
That last one is worth dwelling on. A stale render is indistinguishable from a current one, so it cost us two review rounds where the reviewer reported bugs that were already fixed on disk. Comparing each render's timestamp against every asset it depends on, and marking the ones that are behind, removed the whole class of problem.
One design note: preflight only the ads you are about to render. Checking the whole project means a single unrelated problem blocks every render, which happened to us three times in one afternoon before we noticed the pattern.
If you take one thing from this: measure the artifact, never the intent. The truncated clips, the mirrored footage and the misspelled brand all shipped because a step recorded what it meant to do and nothing checked what it actually produced.
FAQ
Is Remotion free to use for commercial video ads?
It's free for individuals, non-profits, and for-profit companies with up to 3 employees, and those users can create commercial videos with it. For-profit companies with 4 or more employees need a paid Company License. Check the threshold before you build a pipeline on it, because the point where it starts to matter is usually the point where the pipeline has become load-bearing.
Can Remotion generate video ads automatically from footage?
Remotion renders the video, but it does not decide where to cut your footage or what to say. A working ad pipeline wraps it in three other stages: cutting clips out of raw takes at boundaries where the voice actually lands, transcribing and correcting captions, and checking the output before it ships. Remotion is maybe a third of the code and the easiest third.
How do you add burned-in subtitles to a video in Remotion?
Transcribe the audio to word-level timestamps, usually with a local Whisper build, then render each caption page as a component driven by useCurrentFrame(). Two things matter more than the styling: keep a durable list of corrections that is re-applied after every transcription, because re-cutting a clip re-transcribes it and silently loses hand edits, and check that no caption appears more than about 220ms before the words are spoken.
Should I use Remotion or FFmpeg for generating video ads?
Both, for different jobs. FFmpeg is the right tool for cutting, trimming, loudness normalisation and any per-sample audio work. Remotion is the right tool for the visual layer: captions, motion, brand elements, end cards, and rendering the same ad into several aspect ratios. A practical pipeline shells out to FFmpeg to prepare clips and uses Remotion to compose the ad.
Why do my generated video ads sound like they cut off mid-sentence?
Because the cut point came from the transcript rather than the audio. A transcript tells you where a sentence ends as text, not whether the speaker had finished. English statements close on a terminal fall: pitch declines, level tails off, and a pause follows. If you cut before that lands, the ad sounds interrupted even though every word is present. Measure the pause, the pitch fall and the level decay at the cut point, and require a real pause of at least 0.3 seconds.
How much of a raw interview ends up usable as ad footage?
Less than people expect, and the limiting factor is usually clean endings rather than content. Across 36 takes we found 77 usable segments, but several whole recordings had no point at which the voice landed, because the camera was stopped while the speaker was still talking. Around 17% of the runtime that did work was silence between sentences. Leaving a beat of silence before you stop recording is the cheapest thing you can do to make footage editable.
Sources
- Free licence eligibility and the 3-employee threshold — Remotion License, GitHub
- Compositions, the frame timeline and rendering — Remotion documentation
- Loudness normalisation standard used by the measurement tooling — EBU R 128, European Broadcasting Union
- Weighted prediction error dereverberation — nara_wpe, Paderborn University
- Sound-off viewing share and the view-time lift from captions — Facebook Video Ads: Specs and Tips, Superscale
- Local word-level transcription — whisper.cpp