The Art of encoding with NVIDIA Turing NVENC

Dual-PC streaming with x264 has been the leader in H264 encoding for streaming for years – up until NVIDIA released their new Turing generation. This new generation of GPUs with a brand new encoder brough comparable quality to x264 medium (or better), has next to no impact on gaming (unlike an NDI-based dual-PC setup) and is much more affordable.

Let’s take a look at the necessary changes to get your NVENC encoding to look comparable to x264 medium (or better).

Setting up NVENC (for Streaming)

With modern OBS Studio, you have two options: NVENC NVENC H.264 (new) or StreamFXs NVIDIA NVENC H264/AVC (via FFmpeg). The latter has more options to configure, but both will give you comparable quality to x264 medium – as long as you have a Turing GPU encoder. You can check here to see if your GPU has a Turing encoder – note that the GTX 1650 Super also has a Turing encoder.

NVENC NVENC H.264 (new)

This encoder has a much simpler UI, but can’t reach the full quality that NVENC is capable of. Since it is bundled with OBS Studio and has a simpler UI, it is easier to get started with for beginners. The quality achieved this way is comparable to x264 medium most of the time, and sometimes reaches x264 slower.

The settings for this are quite simple: Set “Rate Control” to “CBR”, “Bitrate” to your target bitrate (“6000 Kpbs” in my case), “Keyframe Interval” to “2”, “Preset” to “Max Quality”, “Profile” to “high”, enable “Look-ahead”, enable “Psycho Visual Tuning”, and finally set “Max B-frames” to “3”.

NVIDIA NVENC H.264/AVC (via FFmpeg)

Optimized Settings

This encoder offers much more configuration options and can actually bring out the full potential of NVENC. However due to this it is much more complicated to use correctly, and is not recommended for beginners. In order to not confuse people with the amount of settings, they are available as an image. Adjust the “Target Bitrate” to what you are capable of uploading, and the “Buffer Size” to twice that in order to get the most stable bitrate possible.

There are some settings which depend on the games being played, such as “Adaptive I-Frames”, “Spatial Adaptive Quantization Strength” and “Look Ahead”. But for the majority of games these settings will perform on a level comparable to x264 veryslow in both PSNR and VMAF. These settings somewhat apply to the previous generation (Pascal) as well, though that generation does not gain much from a “Look Ahead” value above 8 frames.

Setting up the Resolution and Framerate

Depending on the bitrate you are targeting, you have several resolution options. No matter which one you pick, the NVENC result will at minimum be comparable to x264 medium, and often sit just slightly above slow unless the x264 preset simply can’t work under the extreme bitrate restriction. Below is a table for resolutions to choose from based on the bitrate, if you are streaming at 60 FPS:

Resolution
(at 60 fps)
Quality
at 3.5 mbit
Quality
at 6.0 mbit
Quality
at 8.0 mbit
1280×720BAA+
1536×864CBA
1920×1080DCB
2304×1296EDC
2560×1440FED
Perceptual Quality Rating for NVENC based on VMAF and PSNR, rated from A (best) to F (worst). E and F can be considered “unwatchable”, especially if transcoding is involved.

The table slightly changes when you are targeting 30 FPS, favoring higher bitrates more:

Resolution
(at 30 fps)
Quality
at 3.5 mbit
Quality
at 6.0 mbit
Quality
at 8.0 mbit
1280×720AA+A++
1536×864BAA+
1920×1080CBA
2304×1296DDC
2560×1440FED
Perceptual Quality Rating for NVENC based on VMAF and PSNR, rated from A (best) to F (worst). E and F can be considered “unwatchable”, especially if transcoding is involved.

Video Comparison

A picture speaks a thousand words, so a video says even more. Instead of debating which is better, just check for yourself. Below are some video examples of x264 veryslow, the OBS NVENC, and the StreamFX optimized NVENC in both 1920×1080 and 1280×720 at 60 FPS. Any framerate differences or stuttering are from the original footage and not part of the encoding process.

Forza Horizon (Racing)

x264 veryslow (1080p60)
x264 veryslow (720p60)
OBS NVENC (1080p60)
OBS NVENC (720p60)
Optimized NVENC (1080p60)
Optimized NVENC (720p60)

DotA 2 (MOBA/Real-Time Strategy)

x264 veryslow (1080p60)
x264 veryslow (720p60)
OBS NVENC (1080p60)
OBS NVENC (720p60)
Optimized NVENC (1080p60)
Optimized NVENC (720p60)

Celeste (Pixel Art, Platformer)

x264 veryslow (1080p60)
x264 veryslow (720p60)
OBS NVENC (1080p60)
OBS NVENC (720p60)
Optimized NVENC (1080p60)
Optimized NVENC (720p60)

(more video examples will be added once their encoding is done)

Final Words

In the past few years NVIDIA has made massive improvements to their encoder, which has evened the playing field far beyond what was expected. With no need to transfer frames from the GPU to the CPU, and quality comparable to x264 medium (or better), NVIDIAs Turing NVENC is pushing the boundaries of what is possible in a single consumer PC.

Whether you use it or not is entirely up to you however. If you already have a working Dual-PC setup that can achieve x264 medium (or better) quality, then you don’t gain much from moving to Turing NVENC. But if you’re currently stuck on anything below x264 medium, or have a Turing GPU ready to test it out – why not give it a shot?

Bookmark the permalink.