1 FPS burns tokens. Extract frames for LLM by scene change

You hand an MP4 to Claude and get back an unsupported-file-type error. So you do what every quickstart shows: pull frames at 1 FPS and ship all of them, turning a ten-minute screen recording into roughly 180,000 input tokens of near-identical screenshots. The frame rate isn't the problem. The duplicates are.

Quick answer: To extract frames for LLM input, run one FFmpeg pass that selects scene changes instead of sampling on a clock (ffmpeg -i in.mp4 -vf "select='gt(scene,0.3)',mpdecimate,scale=768:-2" -fps_mode vfr frames/%04d.jpg), pull the audio to a 16 kHz mono WAV for transcription, and write a JSON manifest pairing each frame filename with its timestamp so the model can cite when it saw something. That's three FFmpeg invocations plus a transcription step you host and babysit yourself; FFmpeg Micro runs the same pipeline as one API call, or as a tool call from your agent through the MCP server.

Why 1 FPS sampling blows up your token bill

Uniform sampling charges you for redundancy. Gemini bills roughly 258 tokens per frame at default media resolution and 66 at low, about 300 tokens per second of video at 1 FPS with audio included. A ten-minute product demo at 1 FPS is 600 frames, around 155,000 tokens of images at default resolution before you've asked a single question.

Look at what those 600 frames contain. A screen recording of a dashboard walkthrough might have 40 distinct screens. The other 560 are the same pixels with the cursor moved. You paid full price for all of them.

Developers who hit this built their own fix. The claude-video-vision project sits at roughly 1,100 stars because Claude's API takes images, PDFs, and text, not MP4 files. A variant of it runs one FFmpeg scene-detection pass at 0.30 sensitivity with an 8% pixel-diff dedup window and a hard cap of 150 keyframes. A dev.to writeup reports 13% to 45% token savings from dedup plus scene detection alone.

The frame extraction pipeline, step by step

Prepping video for a vision model takes four FFmpeg operations and one small script: pick the frames worth sending, drop the repeats, size them to the model's input ceiling, and pull the words out separately.

Pick frames by scene change, not by clock

FFmpeg's select filter exposes a per-frame scene score from 0 to 1 measuring how different a frame is from the one before. Threshold it and you get keyframes at the cuts instead of on a timer.

mkdir -p frames
ffmpeg -i input.mp4 \
  -vf "select='gt(scene,0.3)',metadata=print:file=frames.txt" \
  -fps_mode vfr -q:v 3 frames/frame_%04d.jpg

metadata=print writes each surviving frame's timestamp to frames.txt for the manifest. Output:

frame:0    pts:0       pts_time:0
lavfi.scene_score=0.000000
frame:1    pts:35235   pts_time:14.68
lavfi.scene_score=0.412887

On FFmpeg 6 and later use -fps_mode vfr. The older -vsync vfr still works but prints a deprecation warning.

A threshold of 0.3 is a reasonable start for talking-head and screen-recording footage. Handheld work needs 0.4 or higher, because every shake registers as a scene change.

Kill near-duplicate frames with mpdecimate

The mpdecimate filter compares each frame against the previous one across an 8x8 block grid and drops it if the difference falls below a threshold. It's the cheapest token saving in the pipeline.

ffmpeg -i input.mp4 \
  -vf "fps=2,mpdecimate=hi=768:lo=320:frac=0.33,scale=768:-2,metadata=print:file=frames.txt" \
  -fps_mode vfr -q:v 3 frames/frame_%04d.jpg

Those three numbers are mpdecimate's defaults spelled out: hi=64*12, lo=64*5, frac=0.33. Raise lo to be more aggressive.

One real limit: mpdecimate only compares adjacent frames. An interview cutting A/B/A/B between two fixed angles keeps every cut, because each frame differs from the one before it even though there are only two distinct images. Catching that needs perceptual hashing outside FFmpeg. For screen recordings and single-camera footage, mpdecimate is enough.

Scale to the model's real input ceiling

A 4K frame wastes tokens, because the model downscales it before it looks at it. Current Claude models accept images up to 2576 pixels on the long edge and bill up to roughly 4,784 tokens for one at that size; older models capped at 1568 pixels and about 1,600 tokens. Gemini's media_resolution setting is the equivalent knob.

For most frame analysis, 768 pixels on the long edge is the sweet spot: UI text stays readable and the per-image token cost stays modest. scale=768:-2 keeps the aspect ratio and forces an even height, which matters if you re-encode the frames into video later.

Pull the audio track out separately

Frames alone lose the words. Whisper and most hosted transcription endpoints want 16 kHz mono PCM:

ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le audio.wav

-vn drops the video stream so you're not decoding pixels you already extracted. A ten-minute clip comes out around 19 MB as 16-bit PCM, fine for a direct upload. For a size-limited automation step, encode to Opus with -c:a libopus -b:a 24k and you'll land near 1.8 MB. That's the same size trap that bites people running video through n8n on constrained memory.

Write a manifest so the model can cite timestamps

Image blocks carry no metadata. Send twelve JPEGs and ask "when does the error appear," and the model can't answer, because nothing says which frame came from which second. Fix that with a manifest, and put the timestamp in the prompt text next to each image.

import json, re

timestamps = []
for line in open("frames.txt"):
    m = re.match(r"frame:(\d+)\s+pts:\d+\s+pts_time:([\d.]+)", line)
    if m:
        timestamps.append(float(m.group(2)))

manifest = {
    "source": "input.mp4",
    "sampling": "scene>0.3 + mpdecimate, 768px long edge",
    "frames": [
        {"file": f"frames/frame_{i + 1:04d}.jpg", "t": round(t, 2)}
        for i, t in enumerate(timestamps)
    ],
    "transcript": "transcript.json",
}

json.dump(manifest, open("manifest.json", "w"), indent=2)

Then build the prompt as an alternating sequence: a text block reading frame_0002.jpg at 00:14.68, the image block, the next text block. Now the model answers with a timestamp instead of "somewhere in the middle." To run the whole pipeline in one process, FFmpeg in Python covers the subprocess calls and stderr parsing.

Uniform sampling vs scene detection: the token math

Take a ten-minute screen recording with about 45 distinct screens and run Gemini's published per-frame rates over it. It's arithmetic, not a benchmark, but the ratio holds for any video where most seconds repeat the one before.

The last row matters for long or heavily-cut footage. A fast-cut three-minute music video can produce 400 keyframes from scene detection alone, worse than uniform sampling. Cap the count: keep the top 150 by scene score and log what you dropped.

Run the same pipeline as an API call or an agent tool

Four FFmpeg passes plus a transcription call is not hard code. It's hard operations: an FFmpeg binary pinned to a version, a box with disk for the intermediate frames, and a job that outruns your function timeout the first time somebody uploads a 90-minute recording. That's the part FFmpeg Micro removes. You submit the job, poll or take a webhook, and download the frames and audio, with no encoder to host and no long-running process to babysit. The docs cover the job semantics, and the MCP server exposes the same operations as a tool call, so a Claude agent can extract its own frames mid-conversation. There's a free tier to test it on real footage.

When to skip frame extraction entirely

Scene detection is the wrong tool when the information lives in the motion rather than the cuts. Counting reps, reading sign language, or checking that a UI animation is smooth need frames at a fixed interval, because the point is what changed between two visually similar frames. Use uniform sampling at 5 or 10 FPS there and accept the token bill.

The other case is a model that ingests video natively. The Gemini API accepts video files through its File API, so on Gemini with short clips you can skip this pipeline and let the model sample. You give up control of frame rate and resolution and pay roughly 300 tokens per second of video. Fine for a 30-second clip, terrible for an hour-long recording. The Claude API takes images, PDFs, and text, so on that side frame extraction isn't optional.

Common pitfalls when extracting frames for an LLM

Broken frame-extraction pipelines fail in one of four ways, and all of them are silent.

FAQ

How many frames per second should I extract for a vision model?

Frame rate is the wrong variable to tune for most footage. Scene detection at a threshold of 0.3 yields 4 to 8 frames per minute on screen recordings and interviews, versus 60 at 1 FPS, for the same content. Use fixed-interval sampling at 5 to 10 FPS only when the task depends on motion between similar frames, like counting reps.

How many tokens does a second of video cost in Gemini?

Gemini bills roughly 300 tokens per second of video at 1 FPS with audio included: about 258 for the frame at default media resolution plus the audio track. Low media resolution drops the frame cost to 66 tokens, cutting a ten-minute video from roughly 155,000 image tokens to about 40,000.

Can Gemini and Claude take an MP4 file directly?

The Gemini API accepts video files through its File API and samples them for you. The Claude API accepts images, PDFs, and text, so an MP4 has to become frames plus a transcript before Claude can see it. That gap is why projects like claude-video-vision exist.

Does frame deduplication actually save enough tokens to matter?

Deduplication saves most on the footage people run through vision models most often. A dev.to writeup measured 13% to 45% savings from pixel-diff dedup plus scene detection, and screen recordings sit at the high end because consecutive seconds are usually identical. On a ten-minute recording at default resolution, a 45% cut is roughly 70,000 tokens per run.

Should frames be JPEG or PNG for an LLM?

Use JPEG at quality 3 (-q:v 3). Vision models bill by decoded image dimensions, not file size, so PNG's lossless encoding buys nothing in accuracy and costs about ten times the bytes on upload. The exception is a frame with tiny UI text where JPEG artifacts blur characters, and the fix there is a larger long edge, not a format switch.

Once the frame, audio, and manifest steps stop living in a shell script, they become one API call for an n8n or Make workflow, or a tool your agent calls. Sign up free and point a job at your first video, or see how the same job pattern handles a whole folder at once.