Your AI Dubbing Workflow Isn't Broken. The Dub Just Runs Long.

You wired up ElevenLabs or Whisper plus OpenAI TTS, got a clean Spanish track back, muxed it over the video, and the last thirty seconds of narration play over a black frame. The audio didn't fail. It just doesn't fit.

Quick answer: In an AI dubbing workflow the translated TTS track almost never matches the source video's duration, so measure both with ffprobe first, then stretch the dub with atempo (chained past 2x) or pad it with apad plus -shortest before muxing. Swap it in with ffmpeg -i video.mp4 -i dub.m4a -map 0:v -map 1:a -c:v copy -c:a aac out.mp4 so the video is never re-encoded. If you'd rather not run ffprobe, tempo math, and a mux step inside every automation run, send the same swap to FFmpeg Micro as one API call.

Conventional wisdom says dubbing is a voice problem: pick a better model, get a better dub. The voices are already good. What breaks is that translated speech has a different natural length than the source, and every dubbing tool hands you an audio file with no obligation to match the video it came from. n8n's template gallery has ready-made dubbing workflows now, including template 12723 (ElevenLabs AI Dubbing into YouTube) and 9197 (Whisper, OpenAI TTS, and FFmpeg). Both end by producing a translated audio file. Neither checks whether it fits.

Measure the source and the dub before you mux anything

Duration comparison is the first step of any dubbing pipeline, and it takes two ffprobe calls that cost nothing:

ffprobe -v error -show_entries format=duration -of csv=p=0 source.mp4
# 184.216000

ffprobe -v error -show_entries format=duration -of csv=p=0 dub_es.mp3
# 201.482000

That's a 17.3 second overhang on a three-minute video, about 9% long. Romance-language dubs of English source commonly land 10% to 25% longer, because the same sentence needs more syllables. German and Spanish stretch, Japanese and Korean often compress.

One caveat on measuring MP3 output from a TTS API: format=duration on a VBR MP3 is estimated from headers and can be off by a second or more. If your pipeline branches on the number, decode once to get the truth:

ffprobe -v error -select_streams a:0 -count_packets \
  -show_entries stream=duration,codec_name,sample_rate,channels \
  -of json dub_es.mp3

Better still, ask the TTS provider for WAV or PCM and measure that. A wrong duration produces a correct-looking atempo value that's silently off.

Stretching, padding, and truncating are three different fixes

The gap between the dub and the video decides the fix, and picking the wrong one is what produces the clipped final word people notice immediately. Three cases cover almost every dub:

The atempo filter changes tempo without shifting pitch, which is why it beats resampling for speech. It accepts values from 0.5 to 100 in recent builds, but the safe, well-behaved range is 0.5 to 2.0, and chaining two instances multiplies them. For a 201.482 second dub over a 184.216 second video, the ratio is 1.0937:

ffmpeg -i source.mp4 -i dub_es.mp3 \
  -filter_complex "[1:a]atempo=1.0937[a]" \
  -map 0:v:0 -map "[a]" \
  -c:v copy -c:a aac -b:a 192k \
  -movflags +faststart out_es.mp4

Speech starts sounding rushed somewhere past 1.15 in my experience, and clearly wrong past 1.25. If your ratio lands above that, the answer isn't a bigger atempo value. It's a shorter translation or per-segment timing.

When the dub comes back short, pad instead of stretching. apad appends silence indefinitely and -shortest cuts the output at the video's length:

ffmpeg -i source.mp4 -i dub_es.mp3 \
  -filter_complex "[1:a]apad[a]" \
  -map 0:v:0 -map "[a]" \
  -c:v copy -c:a aac -b:a 192k -shortest out_es.mp4

Without apad, -shortest ends the file when the audio runs out and you lose video. That single missing filter is the most common way a dubbing job silently truncates a clip.

Per-segment timing from the SRT holds sync that one global stretch can't

A single atempo value makes the total lengths match while leaving the middle out of sync, which is the failure people describe as "it drifts." If your translation came from an SRT (and in a Whisper-based pipeline it did), you already have the cue timings you need to place each line where it belongs.

Generate one TTS file per cue, stretch each to its own cue duration, and place it with adelay:

ffmpeg -i source.mp4 -i seg001.wav -i seg002.wav -i seg003.wav \
  -filter_complex "\
    [1:a]atempo=1.081,adelay=1240|1240[s1]; \
    [2:a]atempo=0.942,adelay=5010|5010[s2]; \
    [3:a]atempo=1.113,adelay=9480|9480[s3]; \
    [s1][s2][s3]amix=inputs=3:dropout_transition=0:normalize=0[dub]" \
  -map 0:v:0 -map "[dub]" \
  -c:v copy -c:a aac -b:a 192k -shortest out_es.mp4

Two details that matter. adelay takes milliseconds per channel, so a stereo segment needs 1240|1240, and giving it one value delays only the left channel. And normalize=0 on amix stops FFmpeg from dividing every input's volume by the number of inputs, which is why segment-assembled dubs otherwise come out quiet.

The per-cue atempo value is just cue length divided by rendered segment length. Clamp it to something like 0.85 to 1.15 in your workflow code, and when a segment can't fit, let it run into the next gap rather than compressing it into gibberish.

Keep the original language as a second audio track

Replacing audio doesn't have to mean discarding it. Muxing both tracks costs nothing extra and saves you re-running the whole job when someone asks for the English version:

ffmpeg -i source.mp4 -i dub_es.m4a \
  -map 0:v:0 -map 1:a:0 -map 0:a:0 \
  -c:v copy -c:a:0 aac -b:a:0 192k -c:a:1 copy \
  -metadata:s:a:0 language=spa -metadata:s:a:1 language=eng \
  -disposition:a:0 default -disposition:a:1 0 \
  -movflags +faststart out_multi.mp4

Order matters: the first -map for a type becomes stream 0 of that type, so mapping the dub before the original makes the dub the default. -disposition makes that explicit rather than implied. If you get Stream map '1:a:0' matches no streams, the dub file has no audio stream index you think it has, which is a different problem with a different fix.

Be honest about playback: MP4 carries multiple audio tracks fine, and QuickTime, VLC, and YouTube all expose the track picker, but HTML5 <video> in most browsers plays only the default track with no UI to switch. If end users need to pick a language in a browser, ship separate files per language or use MKV for archival and generate per-language MP4s for delivery.

Never re-encode the video to change the audio

-c:v copy is the difference between a dubbing job that takes a second and one that takes minutes. Re-encoding a 10-minute 1080p H.264 file with libx264 at the medium preset runs several minutes on a typical laptop and costs a generation of quality. Copying the video stream and encoding only the new audio is bounded by disk speed.

The one time you can't copy is when the video and audio containers disagree, for example when the source is MOV with an unusual edit list. Even then, copy first and check the output with ffprobe before reaching for a re-encode.

This is the step where a dubbing workflow stops being an audio problem and becomes an infrastructure problem: you need FFmpeg installed at a version that has atempo and apad, enough disk for the intermediates, and a runtime that doesn't time out on a 40-minute source file. If you're building this inside n8n, Make, or an agent loop, FFmpeg Micro runs the probe, the tempo fit, and the multi-track mux as one API call with no encoder to host and no long-running job to babysit.

Pitfalls that show up in production, not in testing

The failures below all pass a smoke test on a 15 second clip and break on real content:

Where this approach stops working: FFmpeg can time-align a dub, but it can't align it to the speaker's mouth. If you need visual lip-sync, that's a generative video job, not a filter graph. And if your team needs voice casting, a review UI, and per-take approval, that's a full localization platform, a different category from an API that does the swap. For an automated pipeline that produces a dozen language variants a night, the swap is the whole job.

FAQ

Why is my dubbed audio longer than the original video?

Translated speech usually needs more syllables than the source, so a Spanish or German dub of English narration commonly runs 10% to 25% longer at the same speaking rate. TTS models also render pauses generously. Measure both files with ffprobe -show_entries format=duration and treat the ratio as the input to your fix, not as an error.

Can atempo speed up audio more than 2x?

The atempo filter works reliably between 0.5 and 2.0, and to go past that you chain instances: atempo=2.0,atempo=1.25 gives 2.5x. Chaining is legitimate, but a dubbing job that needs more than about 1.25x total is telling you the translation is too long, and the better fix is a shorter script or per-segment placement.

How do I keep the original language as a second audio track?

Map both audio streams in one ffmpeg command with -map 1:a:0 -map 0:a:0, then set which one plays by default with -disposition:a:0 default -disposition:a:1 0. Tag each with -metadata:s:a:N language=spa so players show a usable label instead of "Track 2."

Does replacing the audio re-encode the video?

Replacing audio does not re-encode video as long as you pass -c:v copy. Only the audio stream gets encoded, so a 10-minute 1080p file re-muxes in about a second instead of the several minutes libx264 would take, with no quality loss on the picture.

Should I stretch the whole dub or time it per subtitle line?

Stretch the whole dub when the gap is under about 10% and the narration is continuous. Time it per line from the SRT when the video has pauses, cuts, or on-screen moments the voice has to hit, since a global stretch fixes total length while letting the middle drift. The per-line approach needs one TTS render per cue, which is the same shape as the burn-in step in translated subtitle pipelines.

Once the probe, the tempo fit, and the multi-track mux are one API call, adding a seventh language to a dubbing pipeline is a loop, not a new piece of infrastructure. Sign up free and run the swap on your next dub.