ffmpegbatch-processingvideo-processing

FFmpeg Batch Processing: How to Process Hundreds of Videos

·Javid Jamae·5 min read

You have 500 videos that need transcoding. Maybe you are resizing them for different platforms. Maybe you are adding watermarks before distributing to clients. Maybe you just need to convert everything from MOV to MP4 because someone on your team insists on recording with QuickTime.

Whatever the reason, you are not going to sit there and run ffmpeg 500 times by hand.

This guide covers three approaches to batch processing with FFmpeg, from basic bash loops to parallel execution to API-based pipelines. Pick the one that matches your scale.

The Bash Loop (Simple, Slow)

The most common approach. Loop through a directory and run ffmpeg on each file:

for file in input/*.mp4; do
  ffmpeg -i "$file" -vf scale=1280:720 -c:v libx264 -crf 23 \
    "output/$(basename "$file")"
done

This works. It processes one video at a time, sequentially. For 10 files, fine. For 500 files on a machine with 16 cores, you are using 6% of your available compute.

The other problem: error handling. If file 247 fails because it has a corrupt header, the loop either stops (if you used set -e) or silently skips it and you find out three hours later when the output directory has 499 files instead of 500.

GNU Parallel (Fast, Fragile)

GNU parallel lets you run multiple ffmpeg processes simultaneously:

find input/ -name "*.mp4" | parallel -j 4 \
  ffmpeg -i {} -vf scale=1280:720 -c:v libx264 -crf 23 \
  output/{/}

The -j 4 flag runs 4 jobs concurrently. On a machine with 8 cores, this is a reasonable default. Push it higher and you risk running out of memory, because each ffmpeg process loads the entire input file into memory for decoding.

Parallel processing introduces new failure modes:

  • Memory pressure. Four concurrent transcodes of 4K video can easily consume 16GB+ of RAM.
  • Disk I/O bottleneck. Reading and writing 4 video streams simultaneously saturates most SSDs.
  • Partial failures. If 2 of 4 parallel jobs fail, your output directory has a mix of completed and incomplete files with no clear indication of which is which.

You can add logging, retry logic, and health checks. But at that point you are building a job queue, not running a bash script.

xargs for Quick Parallel Runs

If you do not have GNU parallel installed, xargs works for simple cases:

find input/ -name "*.mp4" -print0 | xargs -0 -P 4 -I {} \
  ffmpeg -i {} -vf scale=1280:720 -c:v libx264 -crf 23 \
  output/$(basename {})

Same parallel execution, fewer features. No progress tracking, no retry, no job logging. Fine for a one-off batch, not for a production pipeline.

The API Approach (Scale Without Infrastructure)

At some point, the bash script approach breaks down. Maybe you need to process thousands of videos. Maybe you need to run jobs from a web application. Maybe you are tired of debugging memory issues on your local machine.

A video processing API handles the infrastructure for you. You send a request, the API processes the video on its own servers, and you get the result back.

With FFmpeg Micro, a batch job looks like this:

const videos = ["video1.mp4", "video2.mp4", "video3.mp4"];

const jobs = await Promise.all(
  videos.map(video =>
    fetch("https://api.ffmpeg-micro.com/v1/jobs", {
      method: "POST",
      headers: {
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
      },
      body: JSON.stringify({
        input: video,
        output_format: "mp4",
        options: { scale: "1280:720", codec: "libx264", crf: 23 }
      })
    })
  )
);

No memory management. No disk I/O tuning. No infrastructure to maintain. The API auto-scales based on your workload.

When to Use Each Approach

ApproachBest ForLimitations
Bash loopUnder 50 videos, simple transformsSlow, sequential, no error recovery
GNU parallel50-500 videos, local processingMemory limits, needs monitoring, fragile
APIAny volume, production pipelinesRequires network, per-job cost

The decision usually comes down to frequency. If you batch process videos once a quarter, a bash script is fine. If you do it daily or it is part of a product workflow, the infrastructure overhead of managing ffmpeg locally costs more than an API.

Cost Comparison

Processing 1,000 one-minute videos (1080p to 720p transcode):

MethodTimeCost
Bash loop (single core)~8 hours$0 (your machine)
GNU parallel (4 cores)~2 hours$0 (your machine)
Cloud VM (c5.2xlarge)~1 hour~$3.40 (on-demand)
FFmpeg Micro API~30 min~$25 (pay per job)

The API costs more per run. But factor in the engineering time to set up parallel processing, handle failures, manage a cloud VM, and maintain the pipeline over months. At $150/hour engineering time, the API pays for itself after the first batch.

Getting Started

If you are processing under 50 videos occasionally, start with the bash loop. If you need parallel processing, try GNU parallel with conservative concurrency (-j 2 to start).

If you are building a product that processes user-uploaded videos, or you need to batch process at scale without managing infrastructure, try FFmpeg Micro. Sign up for free and run your first job in under a minute.

Ready to process videos at scale?

Start using FFmpeg Micro's simple API today. No infrastructure required.

Get Started Free