A bash loop can't batch compress 500 videos. Use a job queue.

To batch compress videos at the scale of a 500-clip backlog, replace the loop with a job queue. Submit jobs concurrently against a fixed concurrency ceiling. Write a job-id-to-source-path ledger as you submit, poll for status, and re-run only the ids that come back failed. A sequential for-loop over 500 clips is bound by the cores of one machine. Terminal scrollback is not a record of what finished.
Nearly every guide that shows you how to batch compress videos assumes one file and one quality setting. A backlog is a different problem. You have mixed durations, mixed sources, and a failure at clip 312 you find out about on Monday.
A for-loop is bound by one box, and raising -threads won't save the queue
The recipe you'll find everywhere is a shell loop that calls FFmpeg once per file in a folder:
for f in ./backlog/*.mp4; do
ffmpeg -i "$f" -c:v libx264 -crf 23 -preset medium \
-c:a aac -b:a 128k "./out/$(basename "$f")"
done
That works, and the arithmetic is what kills it. Five hundred one-minute clips encoded at three times realtime is about 167 minutes of wall clock. Drop to the medium preset on 1080p source and a single encode often runs at or below realtime. Now the same folder is an eight-hour run, and you have to leave the laptop awake for it.
Raising -threads is the usual next move and it buys less than people expect. FFmpeg's own codec documentation notes that -threads is codec-dependent, and that libx264 selects its own thread count by default (ffmpeg.org). On libx264 the flag is frequently a no-op.
x264's threading also scales sublinearly. The long-running threading discussions on the Doom9 forums show gains tapering well before core count. Extra threads cost CPU per frame without proportional speed.
So you reach for xargs -P 8 or GNU parallel and run eight encodes at once. Now the box thrashes, your editor stutters, and the ceiling is still the cores you own. More to the point, parallel shells give you no record. When the run ends you have a folder of outputs. You don't have a list of which inputs produced them, which errored, and which never started.
The ledger is the part the loop recipes skip
A batch run needs two files on disk before it needs speed. One is a ledger mapping each job id to its source path. The other is a run-start timestamp. Submit, append a line, move on. Everything downstream (progress, reconciliation, retry) reads from those two files instead of from your scrollback.
export FFMPEG_MICRO_API_KEY=sk_live_xxx
RUN_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
CONCURRENCY=20
: > ledger.tsv
submit() {
src="$1"; bytes=$(stat -c%s "$src"); base=$(basename "$src")
up=$(curl -sS -X POST https://api.ffmpeg-micro.com/v1/upload/presigned-url \
-H "Authorization: Bearer $FFMPEG_MICRO_API_KEY" -H "Content-Type: application/json" \
-d "{\"filename\":\"$base\",\"contentType\":\"video/mp4\",\"fileSize\":$bytes}")
url=$(jq -r .result.uploadUrl <<<"$up"); name=$(jq -r .result.filename <<<"$up")
curl -sS -X PUT "$url" -H "Content-Type: video/mp4" --data-binary @"$src" >/dev/null
curl -sS -X POST https://api.ffmpeg-micro.com/v1/upload/confirm \
-H "Authorization: Bearer $FFMPEG_MICRO_API_KEY" -H "Content-Type: application/json" \
-d "{\"filename\":\"$name\",\"fileSize\":$bytes}" >/dev/null
job=$(curl -sS -X POST https://api.ffmpeg-micro.com/v1/transcodes \
-H "Authorization: Bearer $FFMPEG_MICRO_API_KEY" -H "Content-Type: application/json" \
-d "{\"inputs\":[{\"url\":\"gs://$BUCKET/$name\"}],\"outputFormat\":\"mp4\",
\"options\":[{\"option\":\"-c:v\",\"argument\":\"libx264\"},
{\"option\":\"-b:v\",\"argument\":\"$(bitrate_for "$src")k\"},
{\"option\":\"-preset\",\"argument\":\"veryfast\"}]}")
printf '%s\t%s\n' "$(jq -r .id <<<"$job")" "$src" >> ledger.tsv
}
export -f submit
find ./backlog -name '*.mp4' | xargs -P "$CONCURRENCY" -I{} bash -c 'submit "$@"' _ {}
The xargs -P is still there. It now caps how many HTTP submissions are open at once, rather than how many encoders fight over your CPU. FFmpeg Micro's published bulk-shorts Bash script uses the same shape. CONCURRENCY defaults to 20 there, so submissions stay inside the plan's concurrency limit. The exact per-plan ceiling isn't published, so start at 20 and back off if submissions start erroring.
Two queue details matter for a mixed backlog. Inputs longer than two minutes run in a separate long lane on their own server. The 40-minute webinar in the folder can't stall the 45-second clips behind it. Links to uploaded inputs stay valid for 24 hours, so a job that waits in line still runs (docs).
The status response carries queue_position and started_at. That's how you measure the wait instead of guessing at it.
Reconcile from the API, then retry only what failed
Once submission is done, the question is which of the 500 landed. Ask the API rather than counting files in the output folder:
curl -sS -H "Authorization: Bearer $FFMPEG_MICRO_API_KEY" \
"https://api.ffmpeg-micro.com/v1/transcodes?status=failed&since=$RUN_START&limit=100" \
| jq -r '.. | objects | select(has("id")) | .id' | sort -u > failed-ids.txt
join -t $'\t' failed-ids.txt <(sort ledger.tsv) | cut -f2 > retry.txt
wc -l retry.txt # 14
retry.txt now holds 14 source paths. The retry pass is the same submit function pointed at that file instead of the folder. The other 486 are never touched. Swap status=failed for status=completed and the same join tells you what to download. Anything in the ledger that appears in neither list is still processing.
Those 14 usually fail for a handful of reasons: a truncated MOV, an unsupported codec in the source, or a malformed options array. Branching on the error beats blindly re-submitting a file that will fail again. Why your transcode job failed (and what to do in the poll loop) covers the cases.
One CRF across 500 mixed clips gives you 500 different file sizes
CRF is a constant-quality target, not a constant-size one. FFmpeg's own H.264 encoding guide states the relationship plainly (FFmpeg wiki). Raising CRF by about 6 roughly halves the bitrate, and lowering it by 6 roughly doubles it. The resulting size depends entirely on how hard the source is to encode.
Run -crf 23 across a folder of screen recordings, phone footage and drone clips. You'll get outputs spanning an order of magnitude in size, when the whole point was a ceiling.
For a backlog with a size target, key the setting off each clip's own duration:
video kbps = (target_MB x 8192 / duration_seconds) - audio_kbps
A 60-second clip aimed at 25 MB gets roughly 3,285 kbps of video after 128k of audio. A 12-minute clip aimed at the same 25 MB gets 156 kbps. That's a different encode decision entirely, and it tells you to drop the resolution instead. That arithmetic is the bitrate_for function in the submit script above. If you don't have ffprobe on the submitting machine, get video duration and codec without ffprobe installed shows the alternatives.
If the goal is consistency rather than a byte ceiling, the decision is a house spec applied to every input. That's a different shape of job, and transcode every user uploaded video, normalize to one house spec walks through picking the spec. If you're not sure which you're doing, how to compress video: five reasons, five different settings sorts it by why you're compressing.
What 500 clips costs, and why the retry pass is free
The reason you can write a reconcile-and-retry loop without flinching is the meter. FFmpeg Micro bills on the duration of the input video, per second, with no rounding up to the minute. Failed jobs are never billed (pricing). One token buys 10 seconds of input.
Five hundred clips averaging a minute is 30,000 seconds, or 3,000 tokens. That's a $25 top-up on any plan including Free, or it fits inside Pro's 7,000 monthly tokens at $49.99. The 14 retries add nothing, because the 14 failures were never charged.
Compare that to the two managed paths. AWS MediaConvert bills per minute of output in normalized minutes, with resolution, codec and frame-rate multipliers. It also applies a 10-second minimum to every output (AWS pricing). A batch of 500 short clips meets that floor 500 separate times.
Getting a bucket of existing videos through MediaConvert means S3 Batch Operations invoking a Lambda. You stand up a manifest, IAM roles and two buckets before the first encode (AWS tutorial). On Azure, the equivalent is a Batch pool, an FFmpeg application package pushed to every node, and a task collection (Microsoft Learn). Both are orchestrators you then own.
Which backlogs belong in a loop and which belong in a queue
Keep the bash loop when the batch is small, the inputs are uniform, and the box is already idle. Twenty clips from one camera, one preset, overnight on a machine you're not using is a loop. Paying anything for it would be silly.
The loop also wins when the files are huge and local. A folder of 8 GB masters is mostly upload time. Starter caps input at 1,024 MB and Pro at 2,048 MB, so those files don't fit the API at all without a trim or a pre-pass.
Move to a job queue when the batch recurs on a schedule. The same applies when the sources are mixed enough that one setting is wrong for half of them. A queue also pays off when the run is long enough that a partial failure costs you a day of detective work. The fourth case is having no shell to run FFmpeg in, and that one is common.
In n8n, Make or Zapier there's no binary and no xargs. The Execute Command node is disabled by default in n8n v2.0 and later, so an HTTP submit plus a poll is the only shape available.
Stand up your own batch infrastructure when you're already deep in one cloud and the media lives in a bucket measured in terabytes. It makes sense when a platform team maintains the pool anyway. At that scale the orchestrator is justified by everything else it runs.
FAQ about batch compressing a folder of videos
How many compression jobs can I submit at once?
Cap your own concurrency when you batch compress videos in bulk, rather than letting the folder size decide. FFmpeg Micro's published bulk script defaults to 20 concurrent pipelines, which is a sensible starting ceiling. The exact per-plan limit isn't published. Raise it gradually and back off when submissions return errors instead of job ids.
Do I get charged for jobs that fail?
Failed jobs are never billed on FFmpeg Micro, which is what makes a retry pass over the failures safe to automate. Checking status, listing jobs and downloading outputs are free as well. A tight poll loop over 500 job ids costs nothing beyond the input seconds you already paid for.
Can I batch compress videos from n8n or Make without a shell?
Batch compressing videos from n8n or Make works because the whole pipeline is HTTP. Submit each item, store the job id on the item, then poll on a later node or a scheduled run. More than 300 builders run FFmpeg Micro from Make or n8n. The n8n integration page covers the node and the poll pattern.
What happens to my uploads if the queue is backed up?
Links to uploaded inputs stay valid for 24 hours, so a job sitting in the queue still has a readable source when its turn comes. Clips longer than two minutes are routed to a separate long lane on their own server. One 40-minute file in the batch can't delay the short clips queued behind it.
How do I know which of the 500 clips finished?
To tell which of the 500 videos finished, query the jobs list filtered by status and run timestamp. Then join the returned ids against the ledger you wrote at submission time. Anything in the ledger that shows up in neither the completed nor the failed list is still in flight. That's a far more reliable answer than counting files in an output directory.
Fourteen failures out of 500 costing exactly $0 is the detail that makes the reconcile-and-retry pass worth writing. It only holds on a meter that charges input seconds and never charges a failure. Sign up free starts you with 200 tokens, roughly 33 minutes of input. That's enough to batch compress videos from a real slice of the backlog before you decide. No servers to run and no FFmpeg to install, called from your own scripts, from n8n, Make and Zapier, or from your AI agents.
About Javid Jamae
Founder & CEO at FFmpeg Micro
Javid is a software engineer, author, and entrepreneur with over 25 years of professional software development experience across enterprise, startup, and consulting environments. He founded FFmpeg Micro to make video processing accessible to developers through a simple, automation-first REST API.
You might also like

Compress video to a specific size: it's arithmetic, not two-pass
Compress video to a specific size: the bitrate formula, why you leave 5-8% headroom, why two-pass can't run on a job API, and how to verify the output bytes.

How to compress video: five reasons, five different settings
How to compress video by reason: upload ceilings, web delivery, platforms that re-encode, storage cost, and AI model input, with the settings each one needs.

Get video duration and codec in PHP without ffprobe installed
Get video duration in PHP three ways: shell out to ffprobe, use PHP-FFMpeg, or call an API when no binary exists, plus the rotation and $_FILES gotchas.
Ready to process videos at scale?
Start using FFmpeg Micro's simple API today. No infrastructure required.
Get Started Free