The Art of encoding with NVIDIA NVENC

Streaming with more than one PC has been the leader in H.264 encoding for years, but NVIDIAs Turing and Ampere generation has put a significant dent into that lead. The new generation of GPUs with the brand new encoder brought comparable quality x264 medium – if you can find a GPU that is. Let’s take a look at what’s needed to set up your stream for massively improved quality.


The guide has been updated for:
StreamFX v0.10 and OBS Studio 27.0

Setting up NVENC (for Streaming)

Modern OBS Studio has two ways to achieve the expected quality: the built-in NVENC H.264 (new) and the addition from StreamFX called NVIDIA NVENC H.264/AVC (via FFmpeg). Both Options can achieve similar quality to x264 medium, but the latter is able to exceed that and rival x264 medium/slow in various situations. Whichever you pick, both of them support zero-copy encoding, and they’re both valid options for streaming.

Built-In: OBS Studio NVENC H.264 (new)

Image reference for Turing/Ampere

The built-in NVENC option in OBS Studio is by far the simplest option and will give you almost identical quality on Maxwell, Pascal, Turing and Ampere, though Turing and Ampere will make use of the new improvements of the NVENC chip. Maxwell and Pascal users can expect to reach x264 veryfast/faster-like quality, while Turing and Ampere users can expect to hit fast/medium-like quality. Below are the settings you need to set:

OptionMaxwell/PascalTuring/Ampere
PresetQualityMax Quality or Quality
Profilehighhigh
Look-aheadEnabled/CheckedEnabled/Checked
Psycho Visual TuningEnabled/CheckedEnabled/Checked
Max B-Frames1 or 23 or 4
Best Built-in NVENC settings

StreamFX: NVIDIA NVENC H.264/AVC (via FFmpeg)

If you’re new to StreamFX’s NVENC integration, it will most likely overwhelm you with the settings it offers. But thanks to all those settings, you can actually go above the default quality by quite a significant amount. Note that I will only cover critical settings, as other settings like Bitrate, Buffer Size and Key Frame Interval are explained elsewhere.

Image reference for Turing/Ampere
OptionMaxwell/PascalTuring/Ampere
PresetHigh QualityHigh Quality
H264
► ProfileHighHigh
► LevelAutomaticAutomatic
Rate Control Options
► ModeHigh Quality
Constant Bitrate
High Quality
Constant Bitrate
► Two PassDisabledEnabled
► Look Ahead8Any value between 16 and 32
► Adaptive I-FramesDisabledDisabled
► Adaptive B-FramesEnabledEnabled
Adaptive Quantization
► Spatial (AQ)EnabledEnabled
► Spatial (AQ) Strength<Any><Any>
► Temporal (AQ)EnabledEnabled
Other Options
► Maximum B-Frames1 or 23 or 4
► B-Frame References“middle”“middle”
► Zero LatencyDisabledDisabled
► Weighted PredictionDisabledDisabled
► Non-reference P-FramesEnabledEnabled
FFmpeg Options (Optional)
► Custom Settings-refs=#-refs=#
Ideal NVENC Settings for StreamFX’s NVENC

The correct value for -refs= depends on the H.264 Level you have chosen and while higher levels do have more reference frames available to them, the devices supporting this are limited in number. In order to target H.264 Profile Level 4.2, you would have to set -refs=4 at 1920×1080, -refs=6 at 1600×900, and -refs=9 at 1280×720. It is possible to calculate this value with the data and formulas provided in “H.264 Annex A.3.1”.

Setting up Resolution and Framerate to match the Bitrate

It is no secret than H.264/AVC is an outdated codec and that platforms should have paved the way for better codecs a long time ago, but it is the solution that we are stuck with until AV1 is adopted by the masses. So in order to get the best quality out of our stream, we should aim to also set up our stream according to the bitrate and codec we use. Below is the average result of a few hundred thousand tests at various resolutions and bitrates, according to PSNR and VMAF (weighted PSNR 30:70 VMAF):

Resolution3.5mbit
30 FPS
3.5mbit
60 FPS
6.0mbit
30 FPS
6.0mbit
60 FPS
8.5mbit
30 FPS
8.5mbit
60 FPS
640×360668798
960×540546577
1280×720434354
1536×864324344
1600×900323243
1920×1080212233
Rating from 1 to 10 based on VMAF and PSNR, weighted towards producing useful ranges. Tests performed with x264 slow.
A 10 is perfect, 9 is near lossless, 8 is indistinguishable, 7 is high quality, 5 is acceptable quality and 3 is watchable.

Please note that watchable in video encoding means that you can decode information within it with reasonable accuracy, instead of it having turned to full garbage. Higher resolutions than 1920×1080 were omitted from the table as the rows would be filled with values between 0 and 1, which just are not very useful to us.

This means that at 3.5mbit, the highest resolution and framerate for a variety streamer is 1280×720 at 30 FPS, or 960×540 at 60 FPS. The equation shifts slightly for 6.0mbit, where you can either go for 1536×864 at 30 FPS or 1280×720 at 60 FPS. Finally at 8.5mbit you are looking at a maximum resolution and framerate of 1920×1080 at 30 FPS or 1536×864 at 60 FPS.

Final Words

In the past few years NVIDIA has made massive improvements to their encoder, which has evened the playing field far beyond what was expected. With no need to transfer frames from the GPU to the CPU, and quality comparable to x264 medium (or better), NVIDIAs Turing NVENC is pushing the boundaries of what is possible in a single consumer PC.

Whether you use it or not is entirely up to you however. If you already have a working Dual-PC setup that can achieve x264 medium (or better) quality, then you don’t gain much from moving to Turing NVENC. But if you’re currently stuck on anything below x264 medium, or have a Turing GPU ready to test it out – why not give it a shot?

Video Examples

ARMA 3 “Walk through the Jungle” #002

x264 slow

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

OBS Studio 27.x

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

StreamFX v0.10

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

FFmpeg 4.4

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

Black Mesa #001

x264 slow

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

OBS Studio 27.x

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

StreamFX v0.10

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

FFmpeg 4.4

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

Call of Duty Modern Warframe TDM Broadcast

x264 slow

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

OBS Studio 27.x

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

StreamFX v0.10

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

FFmpeg 4.4

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

Forza Horizon 4 #002

x264 slow

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

OBS Studio 27.x

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

StreamFX v0.10

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

FFmpeg 4.4

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

GRIP Combat Racing #007 (at 60 FPS)

x264 slow

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

OBS Studio 27.x

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

StreamFX v0.10

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit

FFmpeg 4.4

1920x1080x60 at 8.5mbit
1920x1080x60 at 6.0mbit
1280x720x60 at 6.0mbit
960×540 at 6.0mbit
640×360 at 3.5mbit
Bookmark the permalink.