ffmpegaws-s3video-pipelines

Stop downloading from S3. Give FFmpeg a presigned URL.

·Javid Jamae·10 min read
Stop downloading from S3. Give FFmpeg a presigned URL.

You have a 400 MB MP4 in S3 and a job that starts with aws s3 cp. That copy is dead time: the worker fills a disk it's about to delete, and you pay egress on every byte even when FFmpeg needs the first 200 KB to pull a thumbnail. FFmpeg has spoken HTTP with range requests for years, so the download step is usually optional.

Quick answer: FFmpeg reads an S3 presigned GET URL as a direct input: ffmpeg -i "https://your-bucket.s3.us-east-1.amazonaws.com/in.mp4?X-Amz-Signature=..." -c:v libx264 out.mp4. An FFmpeg S3 presigned URL input works because FFmpeg's http protocol issues HTTP range requests, so it pulls only the byte ranges it needs instead of the whole object. It holds as long as you quote the URL in the shell, sign it for longer than the job runs, and write output to disk before uploading, since S3 rejects the chunked PUT FFmpeg streams by default. If you'd rather not run an encoder at all, FFmpeg Micro accepts that same presigned GET as a job input and hands back a finished file from one API call.

Reading from S3 without downloading the file

FFmpeg's http protocol is a real seekable transport, not a dumb pipe. When the input is an https:// URL, libavformat opens the connection, reads enough to identify the container, then issues Range: requests to jump around the file. S3 answers ranged GETs with 206 Partial Content, so a presigned URL behaves like a local file with 40 ms of latency on every seek.

That's the whole trick. No S3 plugin, no credentials in FFmpeg, no s3:// support. Presigning moves authentication into the query string, and FFmpeg sees a URL it can range over.

A thumbnail grab that never downloads the full object:

ffmpeg -hide_banner \
  -seekable 1 \
  -multiple_requests 1 \
  -reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 30 \
  -ss 00:00:05 \
  -i "https://my-bucket.s3.us-east-1.amazonaws.com/raw/clip.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA...&X-Amz-Date=20260909T120000Z&X-Amz-Expires=3600&X-Amz-Signature=9f3c..." \
  -frames:v 1 -q:v 2 thumb.jpg

Four flags there matter more than they look. -seekable 1 forces FFmpeg to assume range support instead of autodetecting it (the default is -1). -multiple_requests 1 keeps the TCP connection alive across seeks instead of reconnecting per range. -ss sits before -i so it becomes an input seek that turns into a byte-range jump, instead of an output seek that decodes every frame from zero. Every one of those flags is an input option, so moving any after -i silently changes what it does.

Quote the URL. The signature contains &, so the shell forks your ffmpeg command into a background job and hands FFmpeg a truncated URL, which surfaces as a 403 you'll spend twenty minutes blaming on IAM.

Why a HEAD request against your presigned URL returns 403

SigV4 signs the HTTP method, so a URL presigned for get_object authorizes GET and nothing else. Send HEAD to that same URL and S3 returns 403 SignatureDoesNotMatch, even though the identical URL streams fine over GET.

This bites in the wrapper, not in FFmpeg. Node code that "checks the file exists" before spawning, an n8n HTTP Request node set to HEAD, a Kubernetes readiness probe on the asset, a CDN doing an origin HEAD: all fail while the actual transcode works. It's the most common reason a presigned URL "works in the browser but not in my pipeline."

To validate an object without downloading it, use a one-byte ranged GET:

curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \
  -r 0-0 "https://my-bucket.s3.us-east-1.amazonaws.com/raw/clip.mp4?X-Amz-Signature=..."
# 206 1

A 206 means the signature is valid, the object exists, and S3 will honor ranges. If you need HEAD, presign for head_object separately and carry two URLs.

When moov placement turns a 200 KB probe into a full fetch

MP4 files store their index in the moov atom, and where that atom sits decides how many bytes FFmpeg has to move. With -movflags +faststart, moov is at the head, so a probe reads a few hundred kilobytes and stops. Without it, moov sits after the media data, and FFmpeg has to reach the end of a multi-gigabyte object before it can decode one frame.

On a seekable connection that's survivable: FFmpeg range-requests the tail, parses the index, then seeks back. On a non-seekable one, including anything with -seekable 0 or a proxy that strips Accept-Ranges, FFmpeg reads linearly through the whole object just to reach the index. Your "no download" pipeline quietly became a download plus a transcode.

FFmpeg's default probesize is 5,000,000 bytes and default analyzeduration 5,000,000 microseconds, so a badly interleaved remote file can also stall at analysis before any real work starts. Raising them makes the stall longer, not shorter. Fixing moov placement on ingest is the fix: anything you generate gets -movflags +faststart, and anything from a browser recorder or a mobile upload gets checked before it enters the queue, the same way you'd validate input before trusting an "Invalid data found" error.

Writing the output back: presigned PUT versus write-then-upload

FFmpeg can write to an HTTP URL with -method PUT, but standard MP4 muxing can't: finalizing an MP4 means seeking back to the start to write moov, and an HTTP body is one-way. Fragmented MP4 removes the seek:

ffmpeg -i "$SRC_PRESIGNED_GET" \
  -c:v libx264 -preset veryfast -crf 23 -c:a aac \
  -movflags frag_keyframe+empty_moov+default_base_moof \
  -f mp4 -method PUT \
  "$DST_PRESIGNED_PUT"

Against a plain HTTP server that accepts chunked bodies, that works. Against S3 it doesn't. FFmpeg's HTTP muxer sends Transfer-Encoding: chunked by default (-chunked_post 1), and S3 answers chunked PUTs with 501 Not Implemented. Set -chunked_post 0 and FFmpeg has no Content-Length for a stream it hasn't finished producing, which S3 rejects too. That dead end is why the fluent-ffmpeg issue on piping to an S3 presigned PUT (#925) still gets traffic.

The pattern that holds up in production is asymmetric:

ffmpeg -i "$SRC_PRESIGNED_GET" -c:v libx264 -crf 23 -c:a aac \
  -movflags +faststart -f mp4 pipe:1 \
| aws s3 cp - "s3://my-bucket/out/clip.mp4" --expected-size 500000000

Except +faststart needs a second pass over a seekable output, so piping and faststart are mutually exclusive. Write to /tmp, run faststart, then upload. That gives you a normal non-fragmented MP4, multipart upload with retries, and no 5 GB single-PUT ceiling. Streaming straight to a presigned PUT saves disk you didn't need and costs you retries on a failed upload.

Egress and expiry: the two bills nobody plans for

Streaming from S3 is free only when the compute sits in the same region and goes through an S3 gateway endpoint. Pull the same object to a box outside AWS and you pay the standard internet egress rate, $0.09 per GB for the first 10 TB per month in us-east-1. A 400 MB source is about $0.036 per pass, so ten thousand jobs a month is roughly $360 in transfer before a single CPU second. Every range reopen is also a billed GET request at $0.0004 per 1,000, noise for one file and not for a seek-heavy filter graph across a library.

Expiry is the sharper edge. SigV4 presigned URLs max out at 7 days (604,800 seconds), but that ceiling applies only to long-lived IAM user credentials. Sign with temporary credentials from a Lambda execution role, an ECS task role, or sts:AssumeRole, and the URL dies when the session token does, which for AssumeRole defaults to one hour whatever you passed for X-Amz-Expires. A 90-minute transcode signed inside a Lambda gets a 403 partway through, and -reconnect 1 retries into the same 403 until -reconnect_delay_max gives up. Sign for job duration plus a wide margin, from credentials that outlive the job.

Hand over a URL, get one back

Everything above is work you now own: seek flags, moov placement, a HEAD path that 403s, a PUT path S3 won't take, region-pinned workers to dodge egress, and credential lifetimes that cap your job length. That's a media service, not a video step.

The alternative is to keep the presigned URL and drop the encoder. FFmpeg Micro takes a source URL as job input, runs the operation on managed FFmpeg with no servers to run, and returns a URL for the finished file. Submit a job, poll or take a webhook, download the output. Same S3 bucket, same presigned GET, none of the range-request tuning. The API docs have the exact request shape, and the same job runs from n8n, Make, or Zapier by passing the URL through, the same argument as passing a URL instead of file bytes in n8n.

Common pitfalls

Most failures here look like authentication problems and aren't. Before you touch IAM, check:

  • The URL wasn't quoted, so the shell split it on & and FFmpeg got a URL ending at X-Amz-Algorithm.
  • -ss landed after -i, so FFmpeg decoded from byte zero to your seek point and pulled the whole file.
  • Something in the chain sent HEAD. The GET signature doesn't cover it.
  • The URL went into a filter graph. Commas, colons, and equals signs in the query string break movie=, the concat demuxer list, and subtitles= parsing. Fetch to a local path for filter inputs, or escape hard.
  • You used the global s3.amazonaws.com endpoint for a bucket in another region and got a 307 redirect. Use the regional endpoint.
  • You signed the PUT with a Content-Type and FFmpeg sent a different one, so the signature no longer matches.
  • The URL was minted with role credentials that expire before the job finishes, producing a mid-job 403 that reads like a permissions bug.

If the job dies without producing anything, the stderr line tells you more than the exit code does, the same reasoning behind reading FFmpeg's error text rather than its exit status.

FAQ

Can FFmpeg read directly from S3 without downloading the file?

FFmpeg can read directly from S3 by taking a presigned GET URL as its input, because FFmpeg's http protocol uses HTTP range requests and S3 serves ranged GETs with 206 Partial Content. For operations that touch part of the file, like a thumbnail or a short trim, FFmpeg fetches a few hundred kilobytes instead of the whole object. A full transcode still reads every byte, so the savings are latency and disk, not transfer.

Does FFmpeg support `s3://` URLs?

FFmpeg has no S3 protocol and no AWS credential handling. There's no s3:// scheme to enable and no build flag that adds one. Presigning moves authentication into the query string of a normal HTTPS URL, which is the supported path.

Why does my presigned URL work in the browser but fail in FFmpeg?

A presigned URL that works in a browser and fails in a pipeline is almost always hit with the wrong HTTP method or a mangled query string. SigV4 signs the method, so a GET-signed URL returns 403 on HEAD, and an unquoted URL in a shell command loses everything after the first &.

How long should the presigned URL expiry be for an FFmpeg job?

Set the presigned URL expiry to at least three times your worst-case job duration, and generate it from credentials that outlive the job. The X-Amz-Expires value is capped by the signing credentials, so a URL signed inside a Lambda with a one-hour role session expires in one hour even if you asked for seven days.

Is it cheaper to stream from S3 or download first?

Streaming and downloading cost the same in transfer when the job reads the whole file, since S3 bills per GB moved either way. Streaming wins on ephemeral disk and on partial reads, and both are free when the compute runs in the bucket's region behind an S3 gateway endpoint.

If you'd rather send a presigned URL and get a finished file back than tune seek flags and chase 403s, the free tier is enough to run your real source file through and compare it to what your worker produces now.

About Javid Jamae

Founder & CEO at FFmpeg Micro

Javid is a software engineer, author, and entrepreneur with over 25 years of professional software development experience across enterprise, startup, and consulting environments. He founded FFmpeg Micro to make video processing accessible to developers through a simple, automation-first REST API.

Software EngineeringVideo ProcessingFFmpegCloud ArchitectureAPI DesignAutomation

Ready to process videos at scale?

Start using FFmpeg Micro's simple API today. No infrastructure required.

Get Started Free