FFmpeg Can't Auto Reframe Video to Vertical. Do This Instead

You have a 16:9 interview recording and you need a 9:16 clip for Shorts. Crop the middle and the speaker walks out of frame around the 40-second mark, which is exactly where the quotable line lives. Every guide tells you the crop command; almost none of them tell you where to put the crop window.

Quick answer: FFmpeg has no built-in way to auto reframe video vertical, because FFmpeg has no subject detection. crop=ih*9/16:ih gives you a fixed window and the speaker drifts out of it. What ships without a detection model is a fixed crop offset chosen per scene: crop=608:1080:656:0,scale=1080:1920 for a centered 1080p source, with a different x value for each scene where the speaker isn't centered. To skip the encoder hosting and the per-scene bookkeeping, send the same reframe to the FFmpeg Micro reframing API as one call.

FFmpeg crops to 9:16 perfectly well. It just doesn't know where the person is.

The crop filter takes four numbers and applies them to every frame, forever. It has no concept of a face, a body, or motion. That's the whole gap between FFmpeg and the hosted auto-reframe tools you'll find when you search for one: those tools run an object detector first and feed the coordinates to FFmpeg afterward.

The centered crop is the baseline everything else improves on:

ffmpeg -i input.mp4 \
  -vf "crop=608:1080:656:0,scale=1080:1920:flags=lanczos" \
  -c:v libx264 -crf 20 -preset medium -c:a aac -b:a 128k \
  output.mp4

A 1920x1080 source cropped to 9:16 yields 608x1080 of real pixels, so a 1080x1920 output is an upscale of about 1.78x on the horizontal axis. If your source is 4K, crop 2160*9/16 = 1216 wide instead and you get a true 1080p vertical with pixels to spare. On a 1080p source, consider just shipping 608x1080. TikTok accepts down to 540x960 and the platform re-encodes anyway, which the YouTube Shorts settings post covers in more detail.

A fixed offset picked from the footage beats a centered crop

The crop window's x value is a free parameter and centering it is a guess, not a default. Most interview and talking-head footage puts the subject on a rule-of-thirds line, not dead center, because that's how the camera op framed it. Scrub to a representative frame, note where the speaker's nose sits horizontally, and solve:

x = subject_center_x - (crop_w / 2), clamped to the range 0 to iw - crop_w.

For a speaker at the left third of a 1920-wide frame, that's 640 - 304 = 336. One number, applied to the whole clip, and the speaker is centered instead of shoved to the edge of the vertical frame. This single change fixes more clips than any tracking model does, because most source footage doesn't move much.

Scene splitting turns one bad offset into several good ones

Multi-camera edits and b-roll cutaways break the fixed-offset approach: the wide two-shot needs a different crop window than the close-up that follows it. FFmpeg can find the cut points for you, then you crop each segment on its own.

PySceneDetect does step one better than the scene filter if you're already in Python. Its detect-adaptive mode handles lighting changes that fool a plain frame-difference score, and it emits a CSV of scene boundaries you can loop over directly.

Blurred-background padding is the answer when the subject won't hold still

Padding keeps the entire landscape frame visible and fills the empty top and bottom with a blurred, scaled copy of the same video. Nothing leaves the frame, ever, so there's no crop window to get wrong.

ffmpeg -i input.mp4 -filter_complex \
"[0:v]split=2[bg][fg];\
 [bg]scale=1080:1920:force_original_aspect_ratio=increase,crop=1080:1920,boxblur=luma_radius=20:luma_power=2[blurred];\
 [fg]scale=1080:-2[small];\
 [blurred][small]overlay=(W-w)/2:(H-h)/2" \
-c:v libx264 -crf 20 -c:a copy output.mp4

The foreground lands as 1080x608 in the middle, with 656 pixels of blur above and below. gblur=sigma=25 looks cleaner than boxblur but costs noticeably more CPU per frame, and it's the first thing to swap out when a render starts timing out. Padding is the honest choice for panel discussions, screen recordings, and anything where two people sit on opposite sides of the frame: no crop window holds both of them, and cropping to one means cutting the other out of the conversation entirely.

The tiers compare like this on the footage you're likely to have:

Running the first four tiers across a repurposing pipeline is what FFmpeg Micro's reframing endpoint is for: you post the source URL and the target ratio, pick crop or pad, and get the 9:16 file back with no encoder to host. The Reframe & Format blueprint is the same job as a one-click upload when you just have a file on your desktop rather than a workflow.

Where a detection model plugs in, and why it makes the video shake

A detector gives you one number per frame: the subject's horizontal center. Two open-source projects show the full shape of this. KazKozDev/auto-vertical-reframe chains PySceneDetect for cuts, YOLO for person detection, ByteTrack to keep identities stable across frames, and MediaPipe for face landmarks, then hands coordinates to FFmpeg. kamilstanuch/Autocrop-vertical does a leaner version with YOLOv8 and FFmpeg alone.

Feeding raw per-frame detections straight into a crop offset produces video that looks like it was shot handheld by someone with a tremor. Detector bounding boxes wobble by a few pixels every frame even on a motionless subject, and the crop window faithfully reproduces every wobble. Three fixes, applied in this order:

FFmpeg applies the result through crop's x expression, which is evaluated per frame and can reference t:

ffmpeg -i input.mp4 -vf \
"crop=608:1080:x='656+(512-656)*clip((t-2.5)/0.5\,0\,1)':y=0,scale=1080:1920" \
-c:v libx264 -crf 20 out.mp4

That ramps the window from x=656 to x=512 over half a second starting at t=2.5. Commas inside the expression must be escaped as \, or FFmpeg reads them as filter separators and fails with a parse error. The sendcmd filter can drive the same parameter from a text file, but its changes are stepwise, so put sendcmd timestamps on scene cuts where a jump is invisible and use t-based expressions everywhere else.

Pitfalls that cost the most time

Most failed reframes trace back to four things, none of which are about the crop math. Odd crop widths are first: crop=607:1080 fails against yuv420p because chroma planes need even dimensions, so round ih*9/16 to 608 yourself. Second, a crop offset past iw - crop_w throws an invalid size error rather than clamping quietly.

Third, phone footage carries rotation metadata. A clip shot vertically on an iPhone is often stored as 1920x1080 with a rotate=90 side data entry, and crop operates on the stored dimensions, not the displayed ones. Run ffprobe -show_streams input.mp4 | grep -i rotat before you compute anything.

Fourth, crop before you scale, always. Scaling to 1080x1920 first and cropping after throws away resolution you then have to invent back. When you're upscaling a 608-pixel-wide crop to 1080, a VMAF check tells you whether the result held up or turned to mush, which matters more than the CRF value you picked.

FAQ

Can FFmpeg automatically follow a speaker when cropping to vertical?

FFmpeg cannot follow a speaker on its own. The crop filter has no detection or tracking built in, so subject-aware reframing requires an external model such as YOLOv8 or MediaPipe to produce per-frame coordinates that FFmpeg then applies through crop's x expression.

What crop size do I use to convert landscape video to 9:16?

To convert landscape video to 9:16 from a 1920x1080 source, crop 608x1080 (crop=608:1080:656:0 for a centered window), which is the full frame height at a 9:16 ratio with the width rounded to an even number. From a 3840x2160 source, crop 1216x2160 and scale down to 1080x1920 for a true 1080p vertical with no upscaling.

How do I crop video to vertical without losing the subject when they move?

To crop video to vertical without losing the subject in a moving shot, either split the video at scene changes and apply a separate static crop offset to each scene, or pad the full landscape frame against a blurred background so nothing leaves the frame at all. Per-frame tracking is the third option and needs smoothing to avoid jitter.

Does cropping a 1080p video to 9:16 hurt quality?

Cropping 1920x1080 to 9:16 leaves 608x1080 of original pixels, so outputting 1080x1920 upscales by about 1.78x and softens fine detail like text and hair. Shipping the 608x1080 crop directly avoids the upscale, and every major short-form platform accepts it.

Can I reframe video to vertical inside n8n or Make?

You can reframe video to vertical inside n8n, Make, or Zapier by calling a video API from an HTTP node instead of installing FFmpeg in the container. The workflow posts a source URL and a target aspect ratio, then picks up the finished 9:16 file from a webhook or a poll.

Pick your tier, get the offsets right, and the reframe stops being the step that eats your afternoon. If you'd rather not host an encoder to run it, the free tier covers enough clips to see whether one API call replaces the whole script.