# Building video ads with Remotion: a working guide for agents

Give this file to your coding agent. It is a condensed version of a pipeline that
produced 48 video ads from 24 minutes of phone footage.

Source: https://adside.ai/blog/remotion-video-ads
Last updated: 2026-09-03

**Versions this was built and tested on:** Remotion 4.0.499 (all `@remotion/*`
packages pinned to the same version), `@remotion/media` for video and audio,
`@remotion/install-whisper-cpp` with whisper.cpp 1.5.5 and the `medium.en` model
(about 1.5 GB on first run), Python 3 with `nara_wpe` and `numpy`, ffmpeg 7.
Scope: one English-speaking talking head, phone footage, no B-roll, no music.

**The one rule:** measure the artifact, never the intent. Every expensive mistake
below came from trusting what a step meant to do rather than checking what it
produced.

---

## 0. What Remotion is, and when it earns its place

Remotion renders video from React. A composition is a React component plus a
frame rate, dimensions and a duration; `useCurrentFrame()` tells the component
which frame it is drawing, and the renderer walks the frames and encodes them.

Use it when you need **many variations of a structured video**: the same ad in
three aspect ratios, twelve hooks over one body, forty cuts of one interview.
Do not use it to replace an editor on a one-off brand film. The win is
repeatability, not craft.

**Licence, before you build anything on it:** Remotion is free for individuals,
non-profits, and for-profit companies with **up to 3 employees**. For-profit
companies with **4 or more employees need a paid Company License**, and the
licence is per legal entity (an agency with four people needs it whatever the
client's size). https://github.com/remotion-dev/remotion/blob/main/LICENSE.md

---

## 1. Install and project shape

```bash
npm create video@latest        # interactive; choose the blank template (a human step, or answer the prompts)
npx remotion add @remotion/media @remotion/fonts @remotion/google-fonts @remotion/install-whisper-cpp
npx remotion studio            # live preview at localhost:3000
npx remotion render <compositionId> out/video.mp4
```

Always install Remotion packages with `npx remotion add <pkg>` rather than npm
directly, so versions stay locked together across `@remotion/*`.

The shape we run, with what each script does:

```
src/core/        formats.ts (registry, section 6), ad.ts (defineAd), fonts.ts, brand tokens, motion
src/components/  shared primitives: video, captions, end card, disclaimer
src/templates/   hook-proof-cta, ugc-testimonial, problem-solution, product-demo, stat-listicle
clients/<slug>/  brand.ts, ads/*.ts, caption-fixes.json      ← content, not code
clients/<slug>/footage, /captions                             ← cut clips and their transcripts
out/<client>/<ad-id>/<ad-id>--<format>.mp4                    ← renders
scripts/
  takes.ts, scan.ts, segments.ts   find takes in raw footage, score candidate segments
  speech.ts                        speech regions, F0 with octave correction, level (the cut-point signals)
  cut.ts, tighten.ts               cut a segment on the voice; remove dead air with punch-ins
  encode.ts, dereverb.py           linear loudness, WPE dereverb (python), libx264 crf 21
  roomcheck.ts, mixcheck.ts        RT60 estimate; speech vs music balance
  captions.ts, caption-fixes.ts, sync-captions.ts   whisper.cpp, durable corrections, per-token snap
  voiceover.ts, music.ts, fetch-stock.ts            optional assets
  new-client.ts, new-ad.ts, sync.ts                 scaffolding, composition registration
  preflight.ts, render.ts, qa.ts                    checks before, render, checks after
  review.ts                        local review page for the rendered set
```

An ad is a data file; the template is code:

```ts
export default defineAd<UgcContent>({
  id: "ycad-0007-1-learn-highly-targeted",
  client: "adside-yc",
  template: "ugc-testimonial",
  formats: ["reels", "feed", "square"],
  brand,
  audio: { captions: "clients/adside-yc/captions/yc-0007-1.json", voiceInClip: true, music: null },
  content: {
    clip: "clients/adside-yc/footage/yc-0007-1.mp4",
    clipSeconds: 6.92,
    focusX: 0.5, focusY: 0.45,
    endCard: { headline: "Let us run your ads", button: "Try Adside", seconds: 2.8 },
  },
});
```

Our 48 ads are one clean segment each (2.8 to 54 s, most around 20 s), burned-in
captions, and a 2.8 s branded end card. Fonts: `@remotion/google-fonts` for
Google fonts, `@remotion/fonts` `loadFont()` for local files under `public/`.
When content mixes into the template, ad number thirty is a copy-paste of ad
twenty-nine and nothing can be changed globally.

---

## 2. Remotion rules that fail silently

- `<Video>` and `<Audio>` come from `@remotion/media`; images from `<Img>`.
  Plain `<video>` tags do not sync to the frame timeline.
- Assets live in `public/` and are referenced with `staticFile()`.
- **CSS transitions and animations are nondeterministic.** They run on
  wall-clock time while the renderer screenshots frames, so a frame captures
  whatever state the browser happens to be in. Drive every animation from
  `useCurrentFrame()`.
- Prefer `interpolate()` with an easing curve over `spring()` for ad work: a
  spring's settle time is a physics output, so it cannot be pinned to a
  timestamp in the voiceover.
- `@remotion/media`'s `<Video>` draws to a canvas, so `objectFit` is a **prop**,
  not a style. Set it in `style` and your footage letterboxes silently. There is
  no `objectPosition`; re-crop with ffmpeg.
- Round any computed pixel value before it reaches CSS. Sub-pixel values can
  serialise into exponential notation, which the style parser drops: a `gap`
  of `1.0000000000000002e-5` silently becomes zero and your words touch.

---

## 3. Importing footage: cut on the voice, not on the transcript

This is the part that decides whether the ads feel professional.

A transcript tells you where a sentence ends **as text**. It cannot tell you
whether the speaker had finished. English statements close on a *terminal
fall*: pitch declines across the last syllables, level tails off, and a pause
follows. Cut before that lands and the ad sounds like it was interrupted, even
though every word is present.

Measure three things at any candidate cut point, from the audio:

Decode to mono PCM with ffmpeg (`-f f32le -ac 1 -ar 48000`), frame it at 50 ms
hops with a ~40 ms window, then per candidate point:

1. **Pause**: silence after the cut. Under 0.3 s, you are cutting into speech.
2. **Pitch fall**: median F0 of the closing ~300 ms against the utterance's own
   median. A ratio above ~0.93 means the speaker is still going. Tuned on one
   male voice (110–140 Hz); the ratio is relative so it should transfer, but
   re-tune on your speaker before trusting it.
3. **Level decay**: RMS drop into the cut. Require **≥ 3 dB**.

A pause of 0.7 s or more settles the acoustic test on its own. Below that,
require a pause plus either the pitch fall or the decay. Then run the
punctuation test below regardless; cadence cannot prove the thought was
complete.

**Two traps when you implement this:**

- If your probe point is *inside* a speech region, do not measure the pause to
  the next region: that counts the rest of the sentence the speaker is still saying as
  silence, and a mid-word cut scores as a 1.9-second pause. Return zero.
- Autocorrelation pitch detection reads an **octave low** surprisingly often.
  Everything measured a flat ~82Hz until octave correction; the real voice was
  110–140Hz falling to ~83Hz at a terminal fall. Without the correction every
  ending looks equally flat and unjudgeable. After finding the best lag, look
  for a shorter lag whose correlation is within ~85% of it and prefer that.

**Also check the words, not only the voice.** Cadence proves the speaker
*stopped*; it cannot prove the thought was *complete*. A half-second breath
mid-sentence passes every acoustic test. Require the transcript to end on
terminal punctuation, or at least not on a word that cannot end a clause.

**Trim the head too.** Leading silence is dead air at the top of the ad, and it
also makes Whisper smear the first words backwards across it: 1.7s of room
tone made it place "I believe that" at 0.00s when the speaker starts at 1.71s.
Start the cut ~120ms before the first word.

**Some takes cannot be saved.** If the recording was stopped while the speaker
was still talking, there is no landing anywhere in it. Detect that and use
different footage rather than fading out and hoping.

---

## 4. Audio: where ads quietly get thrown away

### Normalise linearly, never with single-pass loudnorm

ffmpeg's `loudnorm` in single-pass mode is a **dynamic** filter. Two things go
wrong and both are invisible unless you measure the delivered file:

- Its lookahead is not flushed, so clips come out short. Ours lost 360ms, which was the
  end of the last word, while the sidecar still recorded the duration that had
  been *requested*. So every downstream check agreed the cut was clean.
- Being dynamic, it compresses. It lifts the quiet tail of each sentence back
  toward speaking level, erasing the falling loudness that tells a listener a
  thought has finished.

Instead: measure integrated loudness with `ebur128`, compute a fixed gain,
apply it with `volume`, and catch transients with `alimiter` only if needed.
Target −16 LUFS for social delivery.

```bash
# measure (ffmpeg writes these to STDERR): lines look like "I: -24.9 LUFS" and "Peak: -3.1 dBFS"
ffmpeg -i in.mp4 -af ebur128=peak=true -f null - 2>&1 | grep -E "I:|Peak:"
# gain = target - I   →  -16 - (-24.9) = +8.9 dB
# apply linearly; -c:v copy so the video stream is not re-encoded
ffmpeg -i in.mp4 -c:v copy -af "volume=+8.9dB,alimiter=limit=0.75:attack=5:release=50" out.mp4
# limit=0.75 linear ≈ -2.5 dBFS = the -1.5 dBTP ceiling minus 1 dB of true-peak headroom
```

Parse with `/I:\s*(-?\d+(?:\.\d+)?)\s*LUFS/` and take the last match. Meta
publishes no loudness target; −16 LUFS integrated is the house target and the
QA band is −17 to −11 LUFS, true peak under −1 dBTP.

**ffmpeg writes measurements to stderr, not stdout.** Capturing only stdout
silently yields your fallback value; ours produced an "+83dB correction".

**Watch sample peak versus true peak.** `alimiter` clamps sample peaks, but
delivery specs are in true peak, which sits higher because the reconstructed
waveform overshoots between samples. Limiting to exactly −1.5 delivered
−0.7 dBTP. Give the limiter about 1dB of extra headroom.

### Reverb: use the right tool

A room shows up as an audible echo. Reaching for a **spectral denoiser** or a
**downward expander** is the intuitive move and both are wrong:

- A denoiser assumes the unwanted part is stationary. Reverb is a delayed copy
  of the voice itself, so it removes the wrong thing and leaves watery,
  metallic artefacts.
- An expander only ducks the tail *between* words. The room is still under
  every syllable.

Worse, an expander **steepens the level decay by construction**, which is
exactly what an RT60 estimate measures, so it scores brilliantly on the metric
while doing almost nothing audible. Beware any measurement that the processing
can flatter directly.

Use real dereverberation: **WPE** (weighted prediction error) predicts the late
reverberation from the signal's own recent past and subtracts it. What is
removed is by construction a delayed copy of the voice.
https://github.com/fgnt/nara_wpe

It is Python in a Node pipeline. `pip install nara_wpe numpy`, then pipe raw
float PCM through a ~40-line script (STFT, `wpe()`, iSTFT; keep a `delay` of a
few frames so direct sound and early reflections are never candidates):

```bash
ffmpeg -i in.mp4 -f f32le -ac 1 -ar 48000 - \
  | python3 scripts/dereverb.py --rate 48000 \
  | ffmpeg -f f32le -ar 48000 -ac 1 -i - out.wav
```

WPE is not fast: budget roughly real time per clip on a laptop.

RT60 0.70s → 0.30s on our footage, transcript word-identical. After it, keep
processing minimal: a small dip around 300Hz if the room is boxy, a little
presence around 4kHz, gentle 2:1 levelling.

**Do not put a high-pass after it.** A 2-pole IIR rings at its corner, and that
ringing is a decaying low-frequency tail: it put RT60 back up to 0.35–0.40s.
Moving it ahead of WPE did not help either.

WPE will not fix everything. It removes late reverb; it cannot un-smear early
reflections. If the decay drops 20dB in 100ms and then plateaus at a −42dB
noise floor, there is no predictable tail left to subtract and the honest answer
is a quieter room or a closer mic.

### Take the dead air out

Around 17% of our raw runtime was not speech: pauses between sentences. Cutting
every gap over ~0.26s down to ~0.1s made ads noticeably punchier without
touching a word.

Removing a pause from a locked-off talking head leaves a **jump cut**, which
reads as a dropped frame. Change the shot size across the cut: step the framing
between wide and ~6% punched in, instantly rather than eased, and it reads as
an edit instead.

---

## 5. Captions: burned in, and spelled correctly

~85% of social video is watched with sound off, and captions are worth roughly
a 12% lift in view time, so burned-in captions are not optional.

Transcribe locally with `whisper.cpp`: no API key, no upload, no per-minute
cost. Use `medium.en` for anything burned into a frame; `base.en` is where
"paid ads" becomes "Paydads". https://github.com/ggerganov/whisper.cpp

```ts
import { installWhisperCpp, downloadWhisperModel, transcribe } from "@remotion/install-whisper-cpp";
await installWhisperCpp({ to: WHISPER_DIR, version: "1.5.5" });
await downloadWhisperModel({ model: "medium.en", folder: WHISPER_DIR });
execSync(`ffmpeg -i "${input}" -ar 16000 -ac 1 "${wav}" -y`);   // whisper.cpp needs 16 kHz mono WAV
const out = await transcribe({ inputPath: wav, whisperPath: WHISPER_DIR, model: "medium.en", tokenLevelTimestamps: true });
```

### Corrections must be durable

Whisper repeats the same mistakes on the same voice. Fixing the caption JSON by
hand works exactly until the clip is re-cut, at which point it is re-transcribed
and every correction is silently lost. Keep corrections in
`clients/<slug>/caption-fixes.json`, re-applied automatically after every
transcription:

```json
{ "words":   { "Atside": "Adside", "Paydads": "paid ads" },
  "phrases": { "that side": "Adside", "WYSI founder": "YC founder" } }
```

`words` are single-token swaps; `phrases` span consecutive tokens, and a
shorter replacement blanks the trailing tokens rather than deleting them so
caption timing is untouched. Match against the joined sentence (next section).

### Fix at the text level, not the token level

Whisper splits words
across tokens unpredictably, and differently every time:

```
"paid ads"      → " Pay"  "d"    "ads"
"Retargeting"   → "Ret"   "og"   " getting"
"Adside.ai"     → " @"    "site" "."    "ai"
"YC founder"    → " W"    "Y"    "SI"   " f"  "under"
"[BLANK_AUDIO]" → " ["    "BL"   "ANK"  "_"   "AUD" "IO" "]"
```

A rule written as `["@","site"]` never fires if your normaliser strips `@` to an
empty string. Match against the **joined sentence** and map the match back onto
whichever tokens cover it. Tokenisation stops mattering.

Two details that will bite:

- When a multi-word replacement spans tokens, only the first token carries the
  leading space, so `"paid"+"ads"` fuses into `paidads`. Re-introduce the
  separator.
- Never let a denylist match the correct spelling. Ours flagged `Adside` itself
  because `\bads?\s*side\b` matches `adside`, which made every clean caption a
  violation.

Strip bracketed markers generically (`[BLANK_AUDIO]`, `(speaking in foreign
language)`) by finding an opening bracket and blanking through the close.
Enumerating their split forms is hopeless.

### Sync captions to the voice

Target: a caption page appears within **0 to +220ms** of the words. Slightly
late is invisible; early means the viewer reads the line and then waits for it.

Snap **per token against the audio**, not per utterance. Grouping by gaps
between tokens fails because the gaps are what Whisper gets wrong: on one take
the speaker says "We're", pauses 1.6s, then "building", and Whisper stretched
"'re" across the whole pause: in caption time there was no gap at all.
Corrections of about a second are normal, so do not cap the shift too tightly.

### Never let captions leave the safe zone

Compute the size from measurement rather than authoring it. Measure the widest
unbreakable word, cap the font size so it fits the column, count the wrapped
lines, and shrink until the block fits the available height. An authored size
should be a request, not a promise.

One CSS trap: `white-space: pre` preserves the leading space but **forbids line
breaks**. If every break opportunity sits inside such a span, wrapping can never
happen and the text overflows however carefully you computed the line count. Use
`pre-wrap`.

---

## 6. One ad, every format

Keep a format registry with the safe insets per placement, and have both the
renderer and any preview overlay read the same source. Never hardcode a pixel
value: derive sizes from `min(width, height) / 1080` and positions from the
safe area, so a new format costs one registry entry. Ours:

```json
{
  "reels":     { "width": 1080, "height": 1920, "fps": 30, "placements": ["Instagram Reels", "Facebook Reels", "TikTok", "YouTube Shorts"],
                 "safe": { "top": 0.14, "bottom": 0.35, "left": 0.06, "right": 0.06 }, "source": "meta-official" },
  "story":     { "width": 1080, "height": 1920, "fps": 30, "placements": ["Instagram Stories", "Facebook Stories"],
                 "safe": { "top": 0.14, "bottom": 0.20, "left": 0.06, "right": 0.06 }, "source": "meta-legacy-stories; use reels if the asset also runs in Reels" },
  "feed":      { "width": 1080, "height": 1350, "fps": 30, "placements": ["Facebook Feed", "Instagram Feed"],
                 "safe": { "top": 0.06, "bottom": 0.10, "left": 0.05, "right": 0.05 }, "source": "house-rule" },
  "square":    { "width": 1080, "height": 1080, "fps": 30, "placements": ["Feed fallback", "Marketplace", "Audience Network", "LinkedIn"],
                 "safe": { "top": 0.06, "bottom": 0.10, "left": 0.05, "right": 0.05 }, "source": "house-rule" },
  "landscape": { "width": 1920, "height": 1080, "fps": 30, "placements": ["In-stream", "YouTube", "Right column"],
                 "safe": { "top": 0.06, "bottom": 0.12, "left": 0.06, "right": 0.06 }, "source": "house-rule" }
}
```

Meta's 9:16 numbers (14% top, 35% bottom, 6% sides; 40% bottom with a
disclaimer): https://www.facebook.com/business/help/980593475366490. The static
gallery in https://adside.ai/blog/create-ad-designs-claude-design.md section
3.2 uses the same 9:16 fractions; its feed house rules differ (100 px margins),
so pick one for your shop and keep it in one file.

Delivery: H.264 in MP4, 30 fps, AAC audio (`Config.setCodec("h264")`; encode.ts
uses libx264 crf 21, preset medium, yuv420p). Meta accepts MP4/MOV up to 4 GB
and recommends 1440 × 2560 for 9:16 and 1440 × 1800 for 4:5; 1080-wide is
accepted. Output path: `out/<client>/<ad-id>/<ad-id>--<format>.mp4`.
Disclaimers (regulated categories) are a shared primitive with a minimum
on-screen time, positioned from the safe area, and switch the 9:16 bottom
inset to 0.40.

---

## 7. Guardrails: the part that actually saves you

Every rule below exists because something shipped wrong. Make each one a check
that blocks the render, and **deliberately break it once to confirm it fires**.

Check before rendering (`preflight.ts`, on the ads about to render):

- The clip does not end while the voice is still going: re-run the section 3
  probe on the last 700 ms of the file that will render, not on what the cut
  *intended*.
- The transcript ends on terminal punctuation, or at least not on a word that
  cannot end a clause.
- No caption page precedes its speech by more than 220 ms (`MAX_CAPTION_LEAD_MS`).
- No wrong brand spelling anywhere in the captions (denylist that must never
  match the correct spelling).
- Every referenced asset exists.

Check after rendering (`qa.ts`):

- Integrated loudness between −17 and −11 LUFS, and consistent **across the
  set** (in a review session the ads play back to back; a 15 LU step between
  them is worse than either level).
- True peak under −1 dBTP.
- No render older than the footage it was built from. Re-cutting a clip does
  not re-render the ads that use it, and a stale render is indistinguishable
  from a current one. This one cost us two review rounds where the reviewer
  reported bugs that were already fixed on disk.

**Preflight only the ads you are about to render.** Checking the whole project
means one unrelated problem blocks every render.

---

## 8. Speed, when you have dozens of ads

Three things gave the largest wins, none of which touch quality:

1. **Cache whole-file analysis.** Every cut was decoding the entire source
   twice. With twelve clips from one recording that is twenty-four full decodes
   of the same file. Cache on path + size + mtime.
2. **Memoise expensive per-frame analysis.** Our out-point search re-ran
   autocorrelation pitch detection at every 0.05s step: about 1.5 billion
   operations per cut, recomputing the same frames hundreds of times.
3. **Deduplicate shared work.** Twenty multi-cut ads referenced sixty segments,
   but only twenty-four were distinct, because a good hook gets reused. Cut each
   distinct piece once and join with the concat demuxer
   (`ffmpeg -f concat -safe 0 -i list.txt -c copy out.mp4`). Precondition: every
   segment was encoded with identical codec parameters, or `-c copy` produces a
   broken file. Cache on content hash, not path + mtime, if more than one
   machine touches the footage.

Then run cuts and renders in parallel, sized to your performance cores, and
render your primary placement first so there is something to review while the
rest encode.

---

## 9. A note on ids and filenames

Derive composition ids from something stable, like the clip name. Ids assigned
by list position collide the moment an item is dropped and the list shifts: and
Remotion refuses to bundle *anything* when two compositions share an id, so
every render fails at once with an error that points nowhere near the cause.

Put a short description in the filename alongside the number. Pick the words
that distinguish each ad from the others rather than the ones they all share:
"YC", "founder" and "paid ads" appeared in nearly every one of ours and carried
no information.

---