High Quality Recording with NVIDIA NVENC

The NVENC encoder on NVIDIA GPUs offers two ways to achieve the quality targets: Constant Quantization Parameter (CQP) and Variable Bitrate Constant Quality (VBR-CQ). Modern NVIDIA GPUs are also easily able to handle both 4:4:4 and 4:2:0 recording at 2560×1440 with more than 60 fps, with zero impact on framerate.

When using VBR-CQ it should be noted that the quality you end up getting is directly related to the settings used which may have an impact on framerate, unlike with CQP. It achieves a similar goal to CRF, but it does not perform the same work and results between NVENC and x264 will differ. Additionally VBR-CQ may produce smaller files than CQP, while occasionally also producing better quality than CQP would.

True Lossless

In H.264 and H.265, exact true lossless is not possible, however you can get extremely close to it with I-only encoding. To maintain the highest possible quality, it is required to change the Color Format to I444 or RGB, match up the Color Space with the content being recording, and Color Range must be Full.

Effective Lossless

Effective lossless requires that you set the Color Format to I444, the Color Range to Full, and the Color Space to match the footage you are trying to capture – most of the time this is sRGB, sometimes it is Bt.709. Unfortunately this means that you can’t use zero-copy encoding until StreamFX adds support for it, which shouldn’t be an issue for Console+PC recording, or Dual-PC recording. Except for the HEVC block, the settings here can be used by H.264/AVC as well.

Visually Lossless / Indistinguishable

Visually lossless still depends on I444 being used, so NV12 is still out of the picture.

High Quality

As we are already below the reasonable quality level where it matters, it is fine to pick NV12 at this point. Note that sharpness of elements will drastically suffer from the subsampling.

Acceptable Quality

You will most likely not be able to reach reasonable quality with this anyway, so you can freely choose between I444 and NV12. Choose the latter if you wish for zero-copy encoding as well as much smaller files.

Additional Information

  • Some settings can be adjusted on a case-by-case basis, as the settings shown here are a “best average quality” kind. The affected settings are:
    • Look Ahead: Can be set as low as 16, but should not be set below it, as you will needlessly restrict quality at that point.
    • Spatial AQ Strength: Some content may look better with higher settings, other content with lower settings.
    • Maximum B-Frames: Fast paced games barely have any use for 4 B-Frames and do much better with 2 or 3. In some extreme cases, even 1 will work fine.
  • The use of VBR-CQ might require additional custom settings to force the encoder to use the maximum number of reference frames:
    • H.264: -refs 32
    • H.265: -refs 4
  • The Keyframe Interval can be freely adjusted, and is not a necessary setting. You’re free to use as high as 5 seconds here, the only reason I’ve set it to 0.5 seconds is seek time in editing software.

Back to the Guide