Batch Processing at Scale

There is no batch endpoint

When you need to process a hundred videos, the instinct is to look for a bulk upload endpoint that takes an array. There isn't one, and you don't need it. Batching is client-side orchestration: you submit N normal jobs against the same single-job endpoints from Lesson 3, but you cap how many are in flight at any moment. That cap is the whole trick. It keeps you under your plan's rate limits and gives you a natural place to isolate failures so one bad input doesn't sink the rest.

This lesson assumes you already have createTranscode, waitForJob, and getDownloadUrl from Lesson 3. We'll just call them.

A concurrency limiter

The core helper is a small pool. It keeps a shared queue of work and spins up a fixed number of workers that each pull the next item until the queue is empty. Every item is wrapped in try/catch and stored as { ok, value } or { ok: false, error }, so a single rejection never takes down the batch:

// batch.mjs
export async function processBatch(items, worker, concurrency = 2) {
  const results = []
  const queue = [...items.entries()]

  async function next() {
    const entry = queue.shift()
    if (!entry) return
    const [i, item] = entry
    try {
      results[i] = { ok: true, value: await worker(item) }
    } catch (e) {
      results[i] = { ok: false, error: e.message }
    }
    return next()
  }

  await Promise.all(Array.from({ length: concurrency }, next))
  return results
}

items.entries() pairs each item with its original index, so results stays aligned with the input order even though workers finish out of order. Raising concurrency starts more workers; lowering it throttles you back under a rate limit.

Transcoding a list of videos

The worker is just the Lesson 3 lifecycle for one item: create the job, wait for it, then return whatever you need downstream. Here we transcode an array of input URLs at a concurrency of 3 and return the finished job's id, status, and a signed download URL:

// run-batch.mjs
import { processBatch } from './batch.mjs'
import { createTranscode, waitForJob, getDownloadUrl } from './client.mjs'

const inputs = [
  'https://example.com/clip-1.mp4',
  'https://example.com/clip-2.mp4',
  'https://example.com/clip-3.mp4',
  'https://example.com/clip-4.mov',
  'https://example.com/clip-5.mp4',
]

const results = await processBatch(
  inputs,
  async (url) => {
    const job = await createTranscode({
      inputs: [{ url }],
      outputFormat: 'mp4',
      preset: { quality: 'medium', resolution: '720p' },
    })
    const done = await waitForJob(job.id)
    const downloadUrl = await getDownloadUrl(done.id)
    return { id: done.id, status: done.status, downloadUrl }
  },
  3, // at most 3 jobs in flight at once
)

const succeeded = results.filter((r) => r.ok)
const failed = results.filter((r) => !r.ok)

console.log(`Done: ${succeeded.length} succeeded, ${failed.length} failed`)
for (const [i, r] of results.entries()) {
  if (r.ok) console.log(`  #${i} ok    ${r.value.id} -> ${r.value.downloadUrl}`)
  else console.log(`  #${i} FAILED ${r.error}`)
}

Run it:

node --env-file=.env run-batch.mjs
# Done: 5 succeeded, 0 failed
#   #0 ok    job_... -> https://...
#   ...

Because results are indexed by input position, results[2] always corresponds to your third URL, even if it happened to finish first or last.

Next up: production patterns for real workloads. Reliable polling, retries with backoff, and cost control.