Drawtext can't do karaoke captions. FFmpeg's ass filter can.

You've seen the style: every word snaps to yellow the moment it's spoken, and the rest of the line sits white behind it. You open FFmpeg, find drawtext, and hit a wall by word three because you'd need one filter instance per word with its own enable='between(t,...)' expression. FFmpeg can absolutely do this. Just not with the filter you reached for.

Quick answer: Word-by-word karaoke captions in FFmpeg come from an ASS subtitle file, not a video filter. You generate an .ass file where each word is wrapped in a {\k<centiseconds>} tag, then burn it in with ffmpeg -i in.mp4 -vf "ass=captions.ass" out.mp4. The \k tag flips each word from the style's SecondaryColour to its PrimaryColour at the right moment, which is exactly the TikTok highlight effect.

Why drawtext is the wrong tool for word-by-word captions

Conventional wisdom says animated text in FFmpeg means drawtext with timing expressions. For a single lower-third or a watermark, that's true. But word-by-word highlighting isn't a text-drawing problem, it's a subtitle-rendering problem, and FFmpeg already ships a subtitle renderer that does karaoke as a first-class feature.

That renderer is libass, the open-source implementation of Advanced SubStation Alpha (ASS). ASS was built for anime fansubbing in the early 2000s, where syllable-timed karaoke was the whole point. The \k tag has been in the format for two decades. You're not building an animation system. You're writing a text file.

Check that your build has it:

ffmpeg -buildconf | grep libass
# --enable-libass

If that returns nothing, the ass filter doesn't exist in your binary and you'll get No such filter: 'ass' at runtime. Homebrew's ffmpeg and the gyan.dev Windows builds include it; some minimal Docker images don't. See how to install FFmpeg if you need a full build.

Step 1: Get word-level timestamps from Whisper

Karaoke captions need a start and end time for every word, not per sentence. Whisper gives you this, but only if you ask.

With the OpenAI Whisper CLI:

whisper audio.wav \
  --model large-v3 \
  --word_timestamps True \
  --output_format json \
  --output_dir ./out

The JSON comes back with a segments array, and each segment has a words array of {word, start, end} objects. Three alternatives:

Raw Whisper's word timings drift by roughly 50-150 ms on fast speech. At normal reading speed nobody notices. On a rapid-fire faceless-channel VO, they do, which is why WhisperX exists.

Step 2: Generate the ASS file with \k karaoke tags

An ASS file has three blocks: [Script Info], [V4+ Styles], and [Events]. The style block controls the look. The events block carries the karaoke timing.

A complete file for a 1080x1920 vertical clip:

[Script Info]
ScriptType: v4.00+
PlayResX: 1080
PlayResY: 1920
WrapStyle: 2
ScaledBorderAndShadow: yes

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Pop,Montserrat,88,&H0000F0FF,&H00FFFFFF,&H00000000,&H80000000,-1,0,0,0,100,100,0,0,1,6,2,2,80,80,340,1

[Events]
Format: Layer, Start, End, Style, MarginL, MarginR, Effect, Text
Dialogue: 0,0:00:00.00,0:00:02.14,Pop,,0,0,0,,{\k32}this {\k28}is {\k54}word {\k40}by {\k60}word

Two things do the work here. PrimaryColour is the highlighted color and SecondaryColour is the color before the highlight arrives. &H0000F0FF is ABGR, so that's alpha 00, blue 00, green F0, red FF, which renders as yellow. SecondaryColour is white. And each {\k32} means "hold this word for 32 centiseconds, then snap it to PrimaryColour."

\k values are in hundredths of a second, and they're cumulative from the line's Start time, not absolute timestamps. That's where most hand-written attempts go wrong.

The Python generator

This reads Whisper's JSON and emits the file. It groups words into lines of at most 4 words or 2 seconds, whichever comes first, the density short-form captions actually use.

import json, sys

HEADER = """[Script Info]
ScriptType: v4.00+
PlayResX: 1080
PlayResY: 1920
WrapStyle: 2
ScaledBorderAndShadow: yes

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Pop,Montserrat,88,&H0000F0FF,&H00FFFFFF,&H00000000,&H80000000,-1,0,0,0,100,100,0,0,1,6,2,2,80,80,340,1

[Events]
Format: Layer, Start, End, Style, MarginL, MarginR, Effect, Text"""

MAX_WORDS, MAX_DUR = 4, 2.0

def fmt(t):
    h, rem = divmod(t, 3600)
    m, s = divmod(rem, 60)
    return f"{int(h)}:{int(m):02d}:{s:05.2f}"

def chunk(words):
    out, line = [], []
    for w in words:
        if line and (len(line) >= MAX_WORDS or w["end"] - line[0]["start"] > MAX_DUR):
            out.append(line)
            line = []
        line.append(w)
    if line:
        out.append(line)
    return out

data = json.load(open(sys.argv[1]))
words = [w for seg in data["segments"] for w in seg.get("words", [])]

print(HEADER)
for line in chunk(words):
    start, end = line[0]["start"], line[-1]["end"]
    parts = []
    for i, w in enumerate(line):
        nxt = line[i + 1]["start"] if i + 1 < len(line) else w["end"]
        dur = max(1, round((nxt - w["start"]) * 100))
        parts.append("{\\k%d}%s " % (dur, w["word"].strip()))
    print(f"Dialogue: 0,{fmt(start)},{fmt(end)},Pop,,0,0,0,,{''.join(parts).strip()}")

Run it: python gen_ass.py out/audio.json > captions.ass.

Note the nxt - w["start"] on line 40. Each word's \k duration runs to the next word's start, not to its own end. That absorbs the silent gaps between words into the preceding highlight, so the yellow never falls back to white mid-line. It also guarantees the \k values sum to exactly End - Start, which is what keeps the last word from snapping early.

Step 3: Burn it in

Use the ass filter, not subtitles:

ffmpeg -i input.mp4 \
  -vf "ass=captions.ass:fontsdir=./fonts" \
  -c:v libx264 -crf 20 -preset medium \
  -c:a copy \
  output.mp4

The subtitles filter routes your file through libavformat's subtitle converter and applies force_style overrides, which can flatten karaoke timing. The ass filter hands the file straight to libass. For anything with \k tags, use ass.

On an M2 MacBook Pro, a 60-second 1080x1920 clip takes about 35 seconds to burn at CRF 20 preset medium. faster-whisper large-v3 on CPU int8 adds roughly 90 seconds on top of that. Call it two minutes per short, end to end.

\k vs one Dialogue line per word

\k recolors. It cannot scale, move, or bounce a word, because a karaoke tag only interpolates between two colors defined in the style. To make a word actually pop larger, you need a different structure.

The per-word approach emits the whole line N times, once per highlight state, each with different inline overrides on the active word. It's what Submagic-style output looks like under the hood. Start with \k, and only move to duplicated lines when a client asks for the scale bounce.

Common pitfalls

The one-call version

Everything above works. It also means you're maintaining a Whisper model, a font directory, an FFmpeg build with libass, and a box big enough to run all three. For a faceless channel pushing 30 clips a day, that's a media microservice you didn't set out to write.

FFmpeg Micro is the FFmpeg API: caption generation and burn-in as a job you submit, with no servers to run and no FFmpeg to install. The job semantics are the same for every operation:

# Submit the job, get an id back, then poll or take the webhook.
# Exact payload fields: https://www.ffmpeg-micro.com/docs
curl -X POST https://api.ffmpeg-micro.com/... \
  -H "Authorization: Bearer $FFMPEG_MICRO_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "input": "https://cdn.example.com/clip.mp4", ... }'

Because it's a plain REST call, the same step drops into n8n, Make, or Zapier with an HTTP node, or into a Claude agent loop through the MCP server. If you're already building a repurposing pipeline, this is the caption stage in turning long-form video into Shorts and Reels or in faceless channel assembly. One caution for n8n users: video jobs outrun the default workflow timeout, so wire up webhooks and polling rather than waiting synchronously.

When to skip all of this

If you're captioning three videos a week by hand, CapCut, Descript, and Submagic will get you there faster than any pipeline, and they give you a preview window while you tune the style. The ASS route pays off when captioning is a step in an automated workflow, when you need the exact same font and color across hundreds of clips, or when a human being is never going to look at the file before it publishes.

It's also the wrong tool if you need selectable captions rather than burned-in ones. Word-level karaoke doesn't survive as SRT or as a YouTube caption track, because neither format has a concept of intra-line timing. Burned pixels are the only way this style ships.

FAQ

Does FFmpeg support karaoke captions natively?

Yes, through libass. FFmpeg's ass and subtitles filters both render Advanced SubStation Alpha files, and the ASS format's \k, \kf, and \ko tags handle syllable-level and word-level highlighting. There's no separate karaoke flag on the FFmpeg command line, and there doesn't need to be.

Why is my karaoke text showing up red before it highlights?

Your SecondaryColour is unset. In ASS, karaoke text renders in SecondaryColour before the \k timer reaches it and PrimaryColour after, and the format's default secondary is red (&H000000FF). Set both colors explicitly in your [V4+ Styles] line.

Can I get the TikTok scale-up pop with \k tags?

No. The \k family only interpolates color. For a scale bounce, emit one Dialogue line per word-highlight state and apply \t(0,120,\fscx120\fscy120) to the active word. It's a bigger file and more generator logic, but it's the only way to animate size.

What's the difference between the subtitles and ass filters?

The subtitles filter converts other formats to ASS first and supports force_style overrides, which can override your karaoke styling. The ass filter passes the file directly to libass with no conversion or style injection. Use ass for karaoke, subtitles for SRT.

Do I need a GPU to run Whisper for word timestamps?

Not for short-form. faster-whisper large-v3 handles a 60-second clip on CPU in about 90 seconds with int8 quantization, which is fine for batch jobs. A GPU or the hosted OpenAI API at $0.006 per audio minute is worth it once you're doing dozens of clips a day.

Try a karaoke burn-in against your own clip in the playground, then sign up free and wire the same call into your workflow.