Run an ONNX Upscaler in FFmpeg 9's DNN Filter Graph

Every FFmpeg upscaling guide written before August 2026 says the same thing: FFmpeg can resize, but it can't invent detail, so buy a desktop AI enhancer. That stopped being true in FFmpeg 9.0, and the guides haven't caught up. What trips people up now isn't the model, it's the build, the tensor layout, and an execution provider that quietly drops to CPU without a warning.

Quick answer: The FFmpeg ONNX DNN filter path in FFmpeg 9.0 is the dnn_processing filter with dnn_backend=onnx, which runs an ONNX super-resolution, denoise, or segmentation model inside the filter graph with no conversion to TensorFlow or OpenVINO first. Running it yourself means a custom FFmpeg build with --enable-libonnxruntime, a matched CUDA 12 and cuDNN 9 stack, and a GPU kept warm at roughly $248 a month for an RTX 4090. The decode, trim, scale, frame-extraction, and encode steps around the model need none of that: FFmpeg Micro runs those as one API call, billed per second of input video.

What FFmpeg 9.0 actually changed about the DNN filters

FFmpeg 9.0 added an ONNX Runtime backend to the DNN filters, so dnn_processing can load a .onnx file directly. Jean-Baptiste Kempf's 9.0 release writeup lists it, and Phoronix reported that the backend came from an AMD engineer and also reaches Ryzen AI NPUs through ONNX Runtime's VitisAI execution provider.

The model-conversion step is gone. Before 9.0, the sr and dnn_processing filters wanted TensorFlow .pb graphs or OpenVINO IR, which is why every tutorial you'll find spends half its length on freezing a SRCNN or ESPCN graph. Those pages are still accurate for 8.x builds and still the top results, so people keep converting ONNX weights they already have.

Four execution providers are exposed on 9.0.1: cpu, cuda, dml for Windows DirectML, and vitisai, with a device_id for picking the GPU. That's the whole surface.

The dnn_processing chain that actually runs

A working ONNX upscale needs three parts: a build that links ONNX Runtime, a pixel format the backend accepts, and the filter invocation. Distro FFmpeg packages don't enable the backend, so the build isn't optional.

Build FFmpeg with --enable-libonnxruntime

Install ONNX Runtime first, then point configure at its headers and libs, which FFmpeg's external libraries documentation covers:

./configure \
  --enable-libonnxruntime \
  --extra-cflags=-I/opt/onnxruntime/include \
  --extra-ldflags="-L/opt/onnxruntime/lib -Wl,-rpath,/opt/onnxruntime/lib"
make -j"$(nproc)"

The rpath matters on Linux. Without it the binary builds fine, then fails to find libonnxruntime.so at runtime, which reads like a missing filter rather than a missing library.

The filter chain

With that build, a 4x super-resolution pass on a 480p clip:

ffmpeg -i input.mp4 \
  -vf "format=rgb24,dnn_processing=dnn_backend=onnx:model=realesrgan_x4.onnx:\
input=input:output=output:backend_configs=execution_provider=cuda&device_id=0" \
  -c:v libx264 -crf 18 -pix_fmt yuv420p upscaled.mp4

The format=rgb24 in front is doing real work. The DNN filters hand the model a frame in whatever format the graph gives them, and a 3-channel model needs a 3-channel frame. Single-channel luma models want format=grayf32 and the classic extractplanes=y / mergeplanes sandwich instead, since the model only sees luma and the chroma planes pass through untouched.

Option names vary by build, so run ffmpeg -h filter=dnn_processing on your own binary before trusting any chain, including this one.

Why your model fails at load time

The ONNX Runtime backend in FFmpeg accepts 4-D NCHW float32 tensors and single-input models only, and it rejects UINT8 or other element types when the model loads, per the dnn_processing option reference.

Three things go wrong in practice. A quantized model exported for mobile carries UINT8 inputs and dies immediately. A model exported in NHWC, the default for anything out of TensorFlow, loads shape-mismatched and either errors or produces garbage. And anything with a second input (a scale factor tensor, a mask, a conditioning vector) is out of scope for this filter.

Check before you debug:

python -c "import onnx; m=onnx.load('model.onnx'); print(m.graph.input)"

You want exactly one entry, elem_type: 1 (float32), and four dims in N, C, H, W order. If the input and output names in that dump aren't input and output, pass the real ones to the filter's input= and output= options. Dynamic height and width dims are what you want for video; a fixed 128x128 input means the filter can only ever see one tile size.

How to tell inference silently ran on CPU

ONNX Runtime falls back from GPU to CPU with no error when the CUDA libraries aren't where it expects them, a known complaint against the runtime: the job works, it just runs far slower. Nothing in FFmpeg's output tells you which provider won.

So measure. Run nvidia-smi dmon in a second terminal while the encode runs. If GPU utilization sits near zero while your CPU fans spin up, the provider silently dropped. Real-ESRGAN users report about 1 to 2 frames per second on CPU, so a 60-second clip at 30 fps is 1,800 frames, which is 15 to 30 minutes of wall clock for one minute of video. If FFmpeg's own fps= readout is in that range, you're on CPU.

Two version traps cause most of these drops. From ONNX Runtime 1.22 onward, only CUDA 12 GPU packages ship, requiring CUDA 12.0 through 12.8, and a runtime built against cuDNN 8.x will not work with cuDNN 9.x. Separately, the default provider priority puts TensorRT ahead of CUDA, and having both onnxruntime and onnxruntime-gpu installed in the same environment collapses the visible provider list to CPU only. Pick one package.

async and nireq have no effect with dnn_backend=onnx. The filter documentation states the ONNX backend runs inference synchronously with a single request, so there's no batching knob to turn. The only lever is a faster device.

The cost math, honestly

Hosting ONNX inference inside FFmpeg is a GPU commitment, not a library choice. An RTX 4090 rents from about $0.34/hr on RunPod Community Cloud and roughly $0.69 to $0.74 on Secure Cloud, which is about $248 a month if you keep one running around the clock. You keep it running because cold-starting a container, pulling weights, and warming the CUDA context costs more latency per job than the inference does. We priced that pattern out in RunPod FFmpeg at $0.33/hr isn't cheap. Price it per video, and the arithmetic doesn't change when the workload is a model.

If you're building an AI video pipeline, inference is one filter in a chain of eight, and it's the only one that needs a GPU. The decode, the trim, the frame extraction for your vision model, the format normalization that feeds the ONNX input, and the final H.264 encode are deterministic FFmpeg work. That's exactly what the FFmpeg Micro API does as a single POST, metered at one token per 10 seconds of input, so pulling a 30-second segment out of a one-hour recording bills 30 seconds. Keep the GPU for the model, send everything else over HTTP, and the box you keep warm gets a lot smaller.

That matters more for agent pipelines than batch jobs. The FFmpeg 9 breakage showing up first in repos like bradautomates/claude-video and 0xchamin/mcptube is frame-extraction code, not inference code. Agents are rarely blocked on the model.

When a plain lanczos scale is still the right call

Running a model is the wrong answer for most upscales, and it was before 9.0 too. If your source is a clean 1080p encode going to 4K for a platform that demands 4K, scale=3840:-2:flags=lanczos plus a light unsharp gets you there in real time with no GPU, no weights, and no hallucinated texture on someone's face. We walked through where that holds in How to Upscale Video with FFmpeg (and When It Actually Helps).

Model inference earns its cost in a narrower band: genuinely low-resolution source, heavy compression artifacts you want removed rather than magnified, or restoration work where the output is the product. Social clips cut from a 1080p master are not that band. Neither is anything you're going to re-encode down to a 9:16 crop afterward.

GPU scaling changed in the same release. scale_npp and the rest of the libnpp filters were deleted in 9.0, so if your pre-model resize ran on NPP it's broken regardless of ONNX. The migration is in scale_npp Removed in FFmpeg 9.

Common pitfalls

The ONNX DNN filter mistakes below cost hours rather than minutes, in rough order of how often they bite:

FAQ

Can I use Real-ESRGAN weights with FFmpeg's ONNX backend as they are?

Real-ESRGAN weights work only if the .onnx export you downloaded is a single-input, 4-D NCHW float32 model with dynamic spatial dims. Many community exports are quantized to UINT8 or carry a second input for the scale factor, and FFmpeg's backend rejects both at load time. Dump the graph inputs with the onnx Python package before you blame the filter.

Do async and nireq speed up dnn_processing with the ONNX backend?

The async and nireq options have no effect when dnn_backend=onnx. FFmpeg's filter documentation says the ONNX backend runs inference synchronously with a single request, so those options are inert here, though they work for other DNN backends.

Is the sr filter still worth using in FFmpeg 9?

The sr filter is now the narrower of the two entry points, since it exists for SRCNN and ESPCN-style super-resolution models and carries its own scale handling. For an ONNX model, dnn_processing is the filter to reach for, because it's the one wired to the ONNX Runtime backend and the execution provider selection.

How do I choose between the CUDA, DirectML, and CPU execution providers?

Pick cuda on Linux with an NVIDIA card and a CUDA 12 runtime, dml on Windows when you want GPU inference without a CUDA install, and cpu only for correctness testing. DirectML runs on AMD and Intel GPUs as well as NVIDIA, which makes it the sane default for a Windows workstation, and it sidesteps cuDNN version matching.

Does FFmpeg Micro run ONNX models for me?

FFmpeg Micro does not load your ONNX weights or run model inference. It runs the deterministic FFmpeg work in the pipeline: transcoding, trimming, scaling, thumbnail and frame extraction, audio extraction, watermarking, captions, and the final encode, over a REST API with a free tier of 200 tokens and no card.

If the part of your pipeline that keeps breaking is frame extraction or the encode rather than the model, that's what you can hand off today. Sign up free, or wire it into an agent loop directly through the MCP server and let Claude or Cursor call the video steps as tools while your GPU box only ever does inference.