FFmpeg NVENC vs libx264: When GPU Encoding Actually Wins

Your transcode queue is backed up, and someone on the team says the fix is a GPU. It might be. NVENC will also quietly hand you worse quality at the same number you used with libx264, and an instance that bills for 24 hours to do 15 minutes of work.

Quick answer: FFmpeg NVENC (-c:v h264_nvenc) moves H.264 or HEVC encoding onto the dedicated encoder chip on an NVIDIA GPU, and it beats libx264 on throughput only when the entire filter graph stays in GPU memory: ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i in.mp4 -vf scale_cuda=1280:720 -c:v h264_nvenc -preset p5 -rc vbr -cq 23 -b:v 0 out.mp4. Two things decide whether it's worth it: NVENC's -cq is not libx264's -crf scale, so equal numbers are not equal quality, and any CPU-side filter forces a copy back to system RAM that erases most of the speedup. If you'd rather not host encoders, pin driver versions in a container, or pay for idle GPU hours, FFmpeg Micro runs the same encode as one API call with no servers to run.

The NVENC command that works, and the default that ruins it

A working FFmpeg NVENC command needs four flags most tutorials leave out, and the most important is -b:v 0. Without it, h264_nvenc falls back to its default target bitrate of 2 Mbps, so your -cq 23 gets clamped and the output looks soft. That's the most common reason people conclude "NVENC quality is bad" after one test.

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 \
  -vf "scale_cuda=1280:720" \
  -c:v h264_nvenc -preset p5 -tune hq \
  -rc vbr -cq 23 -b:v 0 -maxrate 8M -bufsize 16M \
  -c:a copy output.mp4

The libx264 version is shorter, because the CPU encoder has sane constant-quality defaults:

ffmpeg -i input.mp4 -vf "scale=1280:720" \
  -c:v libx264 -preset medium -crf 21 -c:a copy output.mp4

Note the presets. Modern NVENC uses p1 (fastest) through p7 (slowest, highest quality). The old string presets (slow, medium, fast, hq, llhq) still parse but have been deprecated aliases since FFmpeg 4.4. Check what your build supports with ffmpeg -h encoder=h264_nvenc before trusting any flag list online.

NVENC quality vs libx264: CQ 23 is not CRF 23

Most NVENC write-ups treat -cq 23 and -crf 23 as the same setting. Both sit on a 0 to 51 quantizer scale, so the numbers look interchangeable. They aren't. libx264's CRF is a rate factor: the encoder varies quantization frame by frame and block by block using mb-tree lookahead, adaptive quantization, and psychovisual optimizations, targeting constant perceived quality. NVENC's -cq is a quality hint handed to a fixed-function silicon block with a much smaller bag of tricks, and -rc constqp -qp 23 is a fixed quantizer with no adaptation.

Matching quality costs bitrate. NVIDIA positions Turing-generation and later NVENC as landing near x264's medium preset, a real improvement over the Kepler and Pascal encoders, but you spend more bits to get there. If you're replacing an -crf 21 libx264 profile, start around -cq 19 with -b:v 0, then compare file sizes and a few still frames from the same source.

Turn on the features that close the gap. -spatial-aq 1 -aq-strength 8 gives NVENC a crude version of adaptive quantization, -rc-lookahead 20 lets it see ahead before deciding, -multipass qres adds a quarter-resolution first pass on Turing and newer, and -b_ref_mode middle allows B-frames as reference. All four cost throughput, which is the trade: you spend NVENC's speed advantage buying quality back. Two-pass encoding versus CRF covers the same decision on the CPU side.

FFmpeg hardware acceleration only pays if frames never leave the GPU

FFmpeg hardware acceleration is not a single switch, it's a pipeline property, and the failure mode is invisible. Decode on GPU, filter on GPU, encode on GPU, and frames stay in VRAM the whole way. Insert one CPU filter and FFmpeg silently copies every frame across PCIe to system memory and back, which on a 1080p60 source is thousands of round trips per minute.

Two flags control this. -hwaccel cuda alone decodes on the GPU, then downloads frames to system memory, which helps a little. Adding -hwaccel_output_format cuda keeps them in VRAM, which is what you want. Then your filters have to be CUDA filters: scale_cuda, overlay_cuda, bwdif_cuda, yadif_cuda. Ask for a CPU filter on CUDA frames and FFmpeg tells you, obliquely:

Impossible to convert between the formats supported by the filter
'Parsed_drawtext_0' and the filter 'auto_scale_0'

The fix is explicit round-tripping, and writing it out shows what you're paying for:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 \
  -vf "hwdownload,format=nv12,drawtext=text='SALE':x=40:y=40:fontsize=64:fontcolor=white,hwupload_cuda" \
  -c:v h264_nvenc -preset p5 -rc vbr -cq 23 -b:v 0 output.mp4

That command works. It's also frequently slower than running libx264 with the same drawtext: you've added a download and upload of every frame on top of a CPU filter that was already the bottleneck. Burned-in subtitles, drawtext, eq, most libavfilter compositing, and anything from the zscale family live on the CPU. If your pipeline is mostly "transcode this to three sizes," NVENC keeps its advantage. If it's "burn captions, stamp a watermark, add a hook bar," you're building a hybrid graph and the GPU is doing the easy part.

The arithmetic the benchmark posts skip

Benchmark posts report frames per second on one machine. Your bill is decided by utilization, and that number is usually terrible. An AWS g4dn.xlarge with a T4 lists at roughly $0.526 per hour on-demand in us-east-1, about $384 a month left running. Say your pipeline processes three hours of 1080p source per day, and NVENC gets you 8x realtime. That's 22.5 minutes of GPU work per day, about 11 hours a month, against 730 billed hours. You're paying roughly $0.07 per minute of video for hardware that's idle 98.5% of the time.

You can shut the instance down between batches, but GPU AMIs are slow to start: driver initialization plus CUDA context setup pushes cold starts past a minute, before your job downloads a byte. For spiky, event-driven work (a webhook fires, one clip needs transcoding) you're choosing between paying for a warm GPU or eating cold-start latency on every job. Neither is great, and it's the same problem covered in the real cost of self-hosting a video toolkit.

A per-job API changes that math instead of improving it. FFmpeg Micro runs H.264 and HEVC encodes, scaling, watermarks, and captions as a single POST to api.ffmpeg-micro.com with your source URL and output spec, then returns a webhook when the file is ready. No driver pinning, no idle hours, no session ceiling. Work out your effective per-minute number from the arithmetic above, then check it against the pricing comparison; if your GPU sits under about 30% utilization, hourly billing is losing.

Common NVENC pitfalls, especially in containers

Most NVENC problems in production are configuration and environment issues, not encoder quality. Check these before bisecting filter graphs.

Verify the basics first: ffmpeg -hide_banner -encoders | grep nvenc confirms the binary has the encoders compiled in, and nvidia-smi --query-gpu=name,driver_version --format=csv shows what the host has.

When NVENC is the wrong call

NVENC is the wrong tool when quality per bit matters more than throughput, when volume is low or bursty, or when your filter graph is mostly CPU work. The answer depends on your pipeline shape, not on a single benchmark.

Two honest boundaries on our own side. If you need frame-level control over an encoder you own, custom x264 tunings, or a codec build we don't run, a managed API is the wrong layer, keep your own encoders. And if you're already running GPUs at high utilization for other workloads, adding NVENC transcoding is close to free: the encoder chip is separate silicon from the CUDA cores and doesn't compete with them.

FAQ

Is NVENC better than libx264?

NVENC is faster and libx264 produces smaller files at the same visual quality. On a T4 or newer card, NVENC encodes 1080p several times faster than libx264 at -preset medium on typical cloud vCPUs, but it needs more bitrate to match. Pick NVENC for throughput, libx264 when file size or the quality ceiling matters most.

What is the NVENC equivalent of CRF 23?

NVENC has no exact equivalent, because libx264's CRF adapts quantization per frame while NVENC's -cq is a hint to fixed-function hardware. Start with -rc vbr -cq 19 -b:v 0 when replacing -crf 21, then compare output size and a few frames from your own source. Always include -b:v 0, or NVENC caps you at its 2 Mbps default.

Can I use FFmpeg hardware acceleration on AWS Lambda?

AWS Lambda has no GPU, so NVENC is unavailable there and any -c:v h264_nvenc command fails with a missing encoder. Serverless video on Lambda means libx264 inside the 15-minute execution limit, or offloading the encode to an external service. The same constraint applies to Google Cloud Functions, covered in processing video in Firebase.

How many NVENC encodes can I run at the same time?

Consumer GeForce cards allow 3 concurrent NVENC sessions under current drivers. Data-center cards like the T4 and L4 have no session cap and are limited only by their encoder chip's throughput. Exceeding the limit produces OpenEncodeSessionEx failed: out of memory (10), a licensing ceiling rather than a VRAM problem.

Does NVENC work in Docker?

NVENC works in Docker through nvidia-container-toolkit, with the driver supplied by the host and the encode libraries injected into the container at runtime. The catch is version coupling: your FFmpeg build's nv-codec-headers version must be at or below the host driver's supported SDK, so upgrading a host driver or rebuilding an image can break encoding without a line of your code changing.

If you'd rather spend the week on your pipeline than on driver matrices, FFmpeg Micro does H.264 and HEVC transcoding, scaling, watermarks, and captions as one API call from your code, from n8n, Make, and Zapier, or from your AI agents over MCP. Sign up free and run your next encode against the API docs before you provision a GPU.