High Quality Streaming with NVIDIA® NVENC (in OBS Studio)

Streaming with more than one PC has been the leader in H.264 encoding for years, but NVIDIAs Turing and Ampere generation has put a significant dent into that lead. The new generation of GPUs with the brand new encoder brought comparable quality x264 medium – if you can find a GPU that is. Let’s take a look at what’s needed to set up your stream for massively improved quality.

Setting up NVENC (for Streaming)

Modern OBS Studio has two ways to achieve the expected quality: the built-in NVENC H.264 (new) and the addition from StreamFX called NVIDIA NVENC H.264/AVC (via FFmpeg). Both Options can achieve similar quality to x264 medium, but the latter is able to exceed that and rival x264 medium/slow in various situations. Whichever you pick, both of them support zero-copy encoding, and they’re both valid options for streaming.

Built-In: OBS Studio NVENC H.264 (new)

The simplest option is the built-in NVENC encoders which offer less flexibility, but gets you started on the path of quality quicker. Depending on which GPU generation you own, you will be able to reach different peak quality levels: Kepler may reach up to x264 superfast, Maxwell and Pascal may reach up to x264 veryfast, Volta may reach up to x264 faster, and Turing and Ampere may reach up to x264 fast.

Option Kepler Maxwell Pascal Volta Turing Ampere
Preset Medium (P4) up to Slowest (P7)
As with all encoders, higher presets have diminishing returns for much harsher performance impact.
Tune High Quality or Low Latency or Ultra Low Latency
Choose between focusing Quality or (much) lower encoding Latency.
Multipass Mode Two Passes (Full Resolution) or Two Passes (Quarter Resolution) or Single Pass Improves bitrate distribution and often quality.
Profile high or main
High enables additional codec features that improve quality.
Look-ahead Enabled
Enables Look-Ahead as well Adaptive I-Frames and Adaptive B-Frames.
Psycho-Visual Tuning Disabled or Enabled
Enables Spatial Adaptive Quantization at Strength 8 and Temporal Adaptive Quantization.
Max B-frames 0 to 4
May improve quality or reduce quality, depending on the situation. Most gaming content only uses up to 2, while most real world content can use much more. Look-ahead should always be enabled for more intelligent B-Frame placement.
OBS Studio 28.0 and newer only. For empty columns, look at the columns left of it.

StreamFX: NVIDIA NVENC H.264/AVC (via FFmpeg)

With StreamFX's NVENC integration it is possible to edge out a lot of additional quality out of NVIDIA NVENC in many situations. This however comes with a hefty cost in complexity, and requires much deeper understanding of both the H.264 specification and NVIDIA NVENC. Most GPU generations can expect to reach a peak quality two x264 presets higher than with the built-in NVIDIA NVENC encoder. As this option is aimed at people with deeper knowledge of both the standard and NVIDIA NVENC, non-critical settings are omitted.

Option Kepler Maxwell Pascal Volta Turing Ampere
Preset Medium (P4) up to Slowest (P7)
As with all encoders, higher presets have diminishing returns for much harsher performance impact.
Tune High Quality or Low Latency or Ultra Low Latency
Choose between focusing Quality or (much) lower encoding Latency.
H264
Profile High or Main
High enables additional codec features that improve quality.
Level Automatic
Some problematic Driver versions require you to manually set the H.264 Profile Level.
Rate Control Options
Mode Constant Bitrate
Multi-Pass Full Resolution or Quarter Resolution or Single Pass
Improves bitrate distribution and may improve quality.
Look Ahead at least 8 frames
Enables adaptive functionality and may improve temporal quality. Has diminishing returns.
Adaptive I-Frames Enabled or Disabled
Improves quality in many games, but may not work with all streaming services. Requires at least 2 frames of Look Ahead.
Adaptive B-Frames Enabled
Requires at least (1 + Number of B-Frames) frames of Look Ahead.
Adaptive Quantization
Spatial Adaptive Quantization Disabled or Enabled
May improve perceptual quality by favoring smoother surfaces for bitrate distribution.
Spatial Adaptive Quantization Strength 1 (least aggressive) to 15 (most aggressive)
More aggressive behavior means less bitrate available for textured surfaces, resulting in additional noise.
Temporal Adaptive Quantization Enabled
Other Options
Maximum B-Frames 0 to 4
May improve quality or reduce quality, depending on the situation. Most gaming content only uses up to 2, while most real world content can use much more. Adaptive B-Frames should always be enabled for more intelligent B-Frame placement.
B-Frame Reference Mode "Each B-Frame" or "Only B-Frames/2"
Situational, more references may actually reduce quality significantly as more reference slots are used.
Zero Latency Default
Weighted Prediction Default
Non-reference P-Frames Enabled
Reference Frames -1 frames to 16 frames
May improve quality at higher values, but also drastically reduces device compatibility and drastically decreases performance.
Low Delay Key-Frame Scale -1
When Low Latency or Ultra Low Latency tune is used, should either be left at -1 or manually adjusted. Can drastically affect quality and latency.
OBS Studio 28.0 and newer only. For empty columns, look at the columns left of it.
  • The old presets were deprecated by NVIDIA and now map to the new P1-P7 presets, which come with massive performance or quality boosts. High Quality is now Medium (P4), while High Performance is closer to Faster (P2).

Matching your Resolution and FPS with your Bitrate

It is no secret that H.264/AVC is now an outdated codec which should have been replaced long ago. Still we can do something to achieve more quality from NVENC, through the careful choice of resolution, framerate and bitrate. Below are quality metrics based on VMAF for a Canvas Resolution of 2560x1440 with 60 FPS scaled to different Output Resolutions:

Output Resolution 3.5mbit 6.0mbit 8.5mbit
640x360 77.9258 84.1078 87.0414
960x540 77.9592 84.1083 87.0381
1280x720 77.9541 84.1095 87.0325
1920x1080 77.9323 84.0764 87.0543
Average VMAF scores with no enhancement gain over many different files at NVENCs best configuration.
For desktop viewing, a score of 70+ is 'Acceptable Quality', 80+ is 'High Quality', 90+ is 'Indistinguishable', 95+ is 'Effectively Lossless'.
For mobile viewing a score of 60+ is 'Acceptable Quality', 75+ is 'High Quality', 85+ is 'Indistinguishable'.

We can made an okay educated guess at the maximum resolution for each bitrate with the above table. For 3.5mbit the maximum resolution is 960x540, for 6.0mbit the maximum resolution is 1280x720, and for 8.5mbit the maximum resolution is 1920x1080. For some more action packed content it is best to reduce the resolution slightly in order to maintain decent quality, while slower or rarely moving content can sometimes increase the resolution slightly.

Final Words

In the past few years NVIDIA has made massive improvements to their encoder, which has evened the playing field far beyond what was expected. With no need to transfer frames from the GPU to the CPU, and quality comparable to x264 medium (or better), NVIDIAs Turing NVENC is pushing the boundaries of what is possible in a single consumer PC.

Whether you use it or not is entirely up to you however. If you already have a working Dual-PC setup that can achieve x264 medium (or better) quality, then you don’t gain much from moving to Turing NVENC. But if you’re currently stuck on anything below x264 medium, or have a Turing GPU ready to test it out – why not give it a shot?

Comments for: High Quality Streaming with NVIDIA® NVENC (in OBS Studio)