Webhooks for long-running video jobs: reliable pipeline patterns

A 10-minute 1080p transcode doesn't fit inside an HTTP request, so you hand the API a callback URL and move on. Then one webhook never arrives, another arrives twice, and a third lands while your server is mid-deploy. The job finished fine. Your pipeline still says pending.

Quick answer: Webhooks for long running video jobs are a latency optimization, not a delivery guarantee, so treat the webhook as a hint and your job table as the truth. Write a pending job row before you submit, verify the signature and return 200 in under a second, key idempotency on the provider's job ID, and run a reconciler that polls any job still open past its expected duration. With FFmpeg Micro you submit to POST https://api.ffmpeg-micro.com/v1/transcodes, and the completion callback carries the same job record as GET /v1/transcodes/:id, so handler and poller share one code path. Free tier: https://www.ffmpeg-micro.com/auth/signup.

Webhooks don't replace polling, they shorten it

Conventional wisdom says you migrate from polling to webhooks and delete the poller. That works for a while, because webhook delivery on a healthy provider succeeds well over 99% of the time. The failure isn't the delivery rate. It's that a pipeline treats "no event" and "not finished" as the same state.

Polling has a property webhooks cannot: the client drives it. If your consumer was down, polling catches up the moment it returns. A webhook that fired into a 502 is gone once the retry schedule runs out, typically five or six attempts over an hour. Miss that hour and the job is complete on their side and pending on yours forever.

The pattern that survives production is a webhook for speed plus a reconciler for correctness. The webhook cuts median completion latency from one poll interval to a second or two; the reconciler resolves the 0.3% you never see.

Build the pipeline around a job table, not the event

Your own job row is the state machine, and the webhook is one of two inputs that can advance it.

Write the row before you submit

Create the local row first, then call the API, then store the returned ID. Submit first and a crash before the insert leaves a paid transcode with no owner. Write first and the worst case is an orphan row you delete later.

curl -X POST https://api.ffmpeg-micro.com/v1/transcodes \
  -H "Authorization: Bearer $FFMPEG_MICRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [{ "url": "gs://my-bucket/1234567890-video.mp4" }],
    "outputFormat": "mp4",
    "preset": { "quality": "medium", "resolution": "1080p" }
  }'

Pass a URL, not bytes. Uploading a 900 MB file through your own server doubles your egress and puts a multi-minute transfer inside a request that should take 200 ms. If the source lives in S3, hand over a presigned link, the same trick that keeps FFmpeg from downloading the file twice.

Acknowledge in under a second, work afterward

Your handler has one job: validate, persist the raw event, return 200. Network work belongs in a background worker. Download the output inside the handler and the provider times out after 10 or 30 seconds, retries, and a second worker pulls the same file while a third delivery lands.

Return 2xx for anything you've stored, even an event you've seen before. Return 5xx only when you couldn't persist it and want a retry. A 4xx stops most providers permanently: right for a bad signature, wrong for a database blip.

Key idempotency on the job ID, not the event body

Duplicate deliveries are normal: a slow ack, a load balancer timing out after your handler committed, a provider retrying into a region that already succeeded. The fix is a conditional update.

import express from "express";
const app = express();

app.post("/hooks/video",
  express.raw({ type: "application/json" }),   // raw body, see pitfalls
  async (req, res) => {
    if (!verify(req)) return res.status(401).send("bad signature");

    const event = JSON.parse(req.body.toString("utf8"));

    // Idempotent: second delivery of the same jobId+status is a no-op.
    const { rowCount } = await db.query(
      `UPDATE jobs
          SET status = $2, output_url = $3, completed_at = $4
        WHERE provider_job_id = $1
          AND status NOT IN ('completed', 'failed')`,
      [event.jobId, event.status, event.outputUrl, event.completedAt]
    );

    res.sendStatus(200);                        // ack first
    if (rowCount === 1) await enqueueDownload(event.jobId);  // work after
  }
);

The status NOT IN ('completed', 'failed') guard blocks duplicates and out-of-order delivery, the quieter problem. A processing event can arrive after the completed event it preceded, and last-write-wins walks a finished job backward. Rank statuses and only move forward.

Run a reconciler on the jobs that went quiet

The reconciler is a cron job, once a minute, with one query: find every job in queued or processing whose age exceeds its expected runtime plus a margin, and poll GET /v1/transcodes/:id for each. In a healthy week it returns zero rows. In a bad week it's why nobody files a ticket.

Set the margin from real durations. FFmpeg Micro routes jobs into two lanes by input length, and anything over two minutes runs on separate infrastructure so long encodes don't hold up short ones. A 45-second clip and a 40-minute webinar need different thresholds. The status response names the lane:

{
  "success": true,
  "jobId": "job-uuid",
  "status": "completed",
  "outputUrl": "gs://output-bucket/processed-video.mp4",
  "startedAt": "2025-01-01T00:00:12Z",
  "completedAt": "2025-01-01T00:02:30Z",
  "lane": "short",
  "queue_position": 1
}

Because the poll response and the webhook payload describe the same record, both paths can call one applyJobState(record) function. The FFmpeg API overview and docs have the full flow, and the free tier gives you 200 tokens, roughly 33 minutes of video, to test the loop.

Verify the signature against the raw body

Signature verification fails for one reason more than all others: your framework parsed the JSON, you re-serialized it with JSON.stringify, and the bytes no longer match what the sender signed. Capture the raw body before any parser touches it.

import hmac, hashlib, time
from flask import Flask, request, abort

app = Flask(__name__)
SECRET = b"whsec_your_signing_secret"

@app.post("/hooks/video")
def receive():
    raw = request.get_data()                      # bytes, unparsed
    ts  = request.headers.get("X-Webhook-Timestamp", "")
    sig = request.headers.get("X-Webhook-Signature", "")

    if abs(time.time() - int(ts or 0)) > 300:     # 5-minute replay window
        abort(401)

    expected = hmac.new(SECRET, f"{ts}.".encode() + raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):    # timing-safe, never ==
        abort(401)

    store_event(raw)                              # persist before acking
    return "", 200

Three details matter here. Sign the timestamp with the body so a captured request can't be replayed next month. Use hmac.compare_digest or Node's crypto.timingSafeEqual instead of ==, because plain comparison leaks the secret one byte at a time. And use your provider's documented header names, not the placeholders above.

Webhooks and polling compared, per job length

Choosing between webhooks and polling comes down to how long the job runs and how many are in flight.

Under about 30 seconds, polling alone is fine and one fewer moving part. Above two or three minutes, the poll count gets silly and the webhook earns its keep. That's also where long-running AI jobs like dubbing blow past workflow timeouts.

Receiving the callback in n8n, Make, and Zapier

n8n, Make, and Zapier all receive callbacks fine, as long as you split the work into two workflows, since one workflow that waits on a video job hits its execution timeout.

In n8n, workflow one submits the job and stores the returned jobId. Workflow two starts with a Webhook node set to POST, responds immediately, and looks the job up by ID. Use the production URL, not the test URL, which listens for one call and stops. Don't pull the finished video into n8n as binary data; a roughly 1 GB file read into a node can take down a small instance, so pass the URL instead.

In Make, use a Custom Webhook module as the trigger and turn on the data structure so downstream modules see typed fields. Make queues incoming calls, so a burst of 200 completions lands as 200 queued executions instead of dropped requests. In Zapier, Catch Hook returns 200 immediately, so do the download in a later step.

Pitfalls that only show up in production

Video webhook handlers break in a handful of predictable ways once real traffic hits them.

When a webhook is the wrong tool

Webhooks need a stable public HTTPS endpoint, and some environments don't have one. If your consumer is a desktop app, a script behind corporate NAT, or a Lambda you'd rather not expose, poll on a 15-second interval. Same for five clips a day, where a cron poller is less code than signature verification and a replay window. If a user is watching a spinner and the clip finishes in seconds, poll from the browser.

If you drive video work from an agent loop rather than a server, an MCP tool call that returns the job ID and checks status on demand beats a callback nobody is listening for.

FAQ

How do I know when a video is done processing without checking manually?

A video processing API signals completion two ways: it calls a webhook URL you registered at submit time, or you poll its status endpoint until the status flips to completed. Register the webhook for speed and keep a reconciler that polls jobs open longer than expected, since delivery can fail while the job itself succeeds.

What happens if my server is down when the video webhook fires?

Most video APIs retry a failed webhook several times with exponential backoff over roughly an hour, then stop. A job that completed during a longer outage never notifies you again, so polling open jobs by ID after a restart is the only thing that closes the gap.

How do I stop duplicate video processing webhooks from running my pipeline twice?

Key the handler on the provider's job ID with a conditional update that only advances jobs not already in a terminal state. A duplicate delivery matches zero rows and does nothing, and the handler still returns 200 so retries stop.

How do I test video processing webhooks locally?

Expose your local handler with ngrok http 3000 or cloudflared tunnel, register that HTTPS URL as the callback, and run one real job to capture a genuine payload. Replay that saved payload with curl for every later test, so you're not burning processing minutes to debug a JSON parse error.

FFmpeg Micro is built for this shape: submit, get notified, download, no encoder to host and no queue to babysit. Sign up free and wire the handler against a real job today.