FFmpeg 8.0 Whisper Filter: Transcribe Video Without a Whisper API

FFmpeg 8.0 shipped a whisper audio filter, and every release roundup ran the same one-liner: MP4 in, SRT out, no API key. Then you paste the command into your own pipeline and get No such filter: 'whisper'. That gap between the demo and a running caption job is the whole story.

Quick answer: The FFmpeg whisper filter is an audio filter added in FFmpeg 8.0 "Huffman" that runs whisper.cpp inside the filter graph and writes a transcript as text, SRT, or JSON. It only exists in a binary compiled with --enable-whisper, which almost no distro or Homebrew package uses, and it needs a ggml model file you download and host yourself. Running it means owning the build, the model file, and the RAM budget per concurrent job. If you'd rather skip all three, FFmpeg Micro transcribes and burns styled captions in one API call with no servers to run and no model to store, on a free tier.

What the whisper filter added to FFmpeg 8.0

The whisper filter takes an audio stream, feeds it to whisper.cpp, and emits a transcript. It's a real filter in the graph, not a wrapper script, so it composes with everything else FFmpeg does: you can pull audio out of an MP4, resample it, transcribe it, and write an .srt in a single process. It supports the same 99 languages whisper.cpp does, optional GPU inference, and voice activity detection so silence doesn't get hallucinated into words.

What it doesn't do is draw anything. The filter produces a transcript file. Pixels are a separate pass.

The command that turns an MP4 into an SRT

Whisper models expect 16 kHz mono audio, so the resample has to happen before the filter or the graph errors out. This is the working shape:

ffmpeg -i input.mp4 -vn \
  -af "aformat=sample_rates=16000:channel_layouts=mono,whisper=model=/opt/models/ggml-base.en.bin:language=en:queue=3:destination=captions.srt:format=srt" \
  -f null -

A few parts of that are load-bearing. -vn drops the video so you're not decoding frames you'll throw away. format=srt picks the output shape (text and json are the other two). destination is a file path; leave it empty and the transcript goes to stderr as filter metadata instead. -f null - exists because the transcript is a side effect, not the output stream, so there's nothing to mux.

queue=3 is the chunk size in seconds. Smaller chunks mean lower latency and more segment boundaries mid-sentence; larger chunks give whisper more context and better punctuation. Three seconds is a reasonable default for talking-head footage. For a podcast with long unbroken speech, 10 gives noticeably cleaner sentence breaks.

To skip silence, add the Silero VAD model:

-af "aformat=sample_rates=16000:channel_layouts=mono,whisper=model=/opt/models/ggml-base.en.bin:vad_model=/opt/models/ggml-silero-v5.1.2.bin:vad_threshold=0.5:language=en:destination=captions.srt:format=srt"

That's a second model file to download and ship alongside the first one.

Why your FFmpeg build doesn't have the filter

The whisper filter is compile-time optional and off by default, so a stock binary won't have it no matter how new the version string looks. Check before you debug anything else:

ffmpeg -hide_banner -buildconf | grep -- --enable-whisper
ffmpeg -hide_banner -filters | grep whisper

If both come back empty on FFmpeg 8.0 or later, your binary was built without libwhisper. Running the command anyway gives you this:

[AVFilterGraph @ 0x55a3c1e2f480] No such filter: 'whisper'
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument

That message means the same thing as the missing-encoder case in Unknown encoder 'libx264' isn't a bug, it's your FFmpeg build: the feature isn't broken, it isn't in the binary. Debian, Ubuntu, Alpine, and the common Docker images all ship builds without it. Homebrew's ffmpeg formula doesn't enable it either. To get it, you build whisper.cpp first, install it so pkg-config can find it, then configure FFmpeg with --enable-whisper and compile. On a container base image that's a multi-stage build and a few hundred megabytes of toolchain you now maintain across FFmpeg releases.

A second failure shows up after the build is right and the model path is wrong. The filter initializes whisper.cpp, whisper.cpp can't open the file, and you get its own loader error before FFmpeg says anything useful:

whisper_init_from_file_with_params_no_state: failed to load model

Nine times out of ten that's a relative path evaluated from a different working directory inside a container.

Model size versus accuracy versus runtime

The model file is not part of FFmpeg. You download a ggml model from the whisper.cpp model repository on Hugging Face and host it yourself, which means it goes into your image, your volume, or your object storage, and it gets loaded into memory on every job.

Runtime scales roughly with model size, and the jump from base to medium is far more than 2x on CPU. The number that matters for a pipeline isn't single-job speed, it's memory times concurrency. Four simultaneous medium jobs need about 8.4 GB resident before you count the decode buffers, so a 8 GB worker that transcribes one file fine will get OOM-killed the moment your queue depth hits three. On GPU, the same arithmetic applies to VRAM, and a consumer card running large-v3 handles fewer parallel streams than most people assume.

Long files add a second constraint. The filter processes audio in queue-sized chunks as it decodes, so a 90-minute recording holds a worker for the entire transcription. If that worker also serves HTTP requests, you've built a timeout. This is the same operational wall described in The NCA Toolkit is free, self-hosting it isn't cheap: the software costs nothing and the always-on machine underneath it costs every hour.

A transcript is not a burned-in caption

The whisper filter's output is an SRT file, which means plain timings and plain text. Nothing about font, position, outline, or the chunky centered style that social captions actually use. Getting pixels takes a second pass through subtitles or ass:

ffmpeg -i input.mp4 \
  -vf "subtitles=captions.srt:force_style='FontName=DejaVu Sans,Fontsize=26,PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,BorderStyle=3,Outline=2,Alignment=2,MarginV=70'" \
  -c:v libx264 -crf 20 -preset medium -c:a copy captions_burned.mp4

That pass has its own failure mode: force_style names a font, and if fontconfig can't resolve it inside your container, FFmpeg silently substitutes something else and your captions render in a font you never chose. FFmpeg fontconfig error isn't fatal, it ships the wrong font covers why that one is easy to miss in automated runs. So the honest version of "one command from MP4 to SRT" is: one build you maintain, two model files you host, two FFmpeg passes, a font package, and a memory budget.

This is the part FFmpeg Micro collapses. Auto Captions takes the video, transcribes it, and returns the styled burn-in with a review step before the render, as one job you submit and one file you download. No --enable-whisper build, no ggml file in your image, no worker sitting idle between batches.

When owning the model file is the right call

Self-hosting the whisper filter wins in three situations, and they're specific. If the audio can't leave your infrastructure for legal reasons, local inference is the only answer. If you already run GPU workers for other inference and they have idle capacity, the marginal cost of transcription is close to zero. And if you're transcribing continuously at high volume, a machine you already pay for beats per-minute pricing at some crossover point.

Outside those, the math usually goes the other way. An always-on cloud GPU instance in the common $0.40 to $0.60 per hour range is north of $300 a month before storage and egress, and a caption pipeline that runs in bursts leaves most of that idle. Add the build maintenance every time FFmpeg or whisper.cpp releases, and the model file you now version alongside your code.

The class of alternative to weigh against a hosted media API isn't another transcription service, it's a managed queue plus your own GPU pool. That's a real option if you have platform engineers. It is not a weekend project.

Common pitfalls

Most whisper-filter problems come from four places, and the error messages point away from all of them.

FAQ

Does the ffmpeg whisper filter work in a standard apt or Homebrew build?

The whisper filter is missing from nearly every packaged FFmpeg build, including Debian, Ubuntu, Alpine, and Homebrew, because it requires the optional --enable-whisper compile flag and libwhisper. Run ffmpeg -hide_banner -filters | grep whisper to confirm. An empty result means you need to compile FFmpeg yourself against whisper.cpp.

What model file does the FFmpeg whisper filter need?

The whisper filter needs a ggml-format whisper.cpp model that you download and store yourself, passed to the filter with model=/path/to/ggml-base.en.bin. Sizes run from 75 MB for tiny.en to 2.9 GB for large-v3. Voice activity detection needs a second file, the Silero VAD model, passed as vad_model.

Can FFmpeg 8.0 burn the captions in, not just write an SRT?

FFmpeg 8.0 can burn captions in, but not with the whisper filter alone. The whisper filter writes a transcript file, and a second pass through the subtitles or ass filter renders that file onto the video with force_style controlling font, size, outline, and position. Two commands, two decodes of the source.

How long does whisper filter transcription take on a long video?

Transcription time with the whisper filter scales with model size and audio length, and a 60-minute file occupies a worker for the whole run. The filter consumes audio in queue-second chunks as FFmpeg decodes, so there's no way to shortcut a long file on a single machine. Plan for a queue and a webhook, not a synchronous call.

Is the whisper filter accurate enough for social captions?

For clean single-speaker English, base.en produces captions accurate enough for social video with light cleanup, and small.en handles accents and background music noticeably better. Product names, brand spellings, and technical jargon are where every whisper model size still misses, which is why a review step before burn-in matters more than the model you pick.

If your caption step is one node in a bigger chain, the older recipe in Auto-Generate and Burn In Captions with Whisper + FFmpeg shows the same job wired end to end. When you'd rather not own the build, the model file, or the worker, sign up free and send the video as one call instead.