OBS NVENC Error - How to Fix NVIDIA Encoding Issues

OBS NVENC Error: How to Fix NVIDIA Encoding Issues

NVENC is a dedicated encoding chip on your NVIDIA graphics card, separate from the cores that render games. OBS hands it raw frames and gets compressed video back, which is why hardware encoding barely touches your CPU.

When it fails, OBS shows a generic dialog and suggests updating your drivers. That advice is a reasonable default, and it is also why a lot of people clean install their drivers three times and stay broken: NVENC returns a specific error code explaining what it refused to do, and OBS buries that code in the log.

There are five or six distinct causes behind this error, and they have almost nothing to do with each other. Reinstalling drivers fixes one of them. Find your code first.

Step 1: Get the Actual Error Code

1. In OBS, open Help > Log Files > View Current Log.
2. Scroll to the bottom and look for lines beginning [obs-nvenc] or containing NVENC.
3. Read the last such line before the session ended.

You will find one of the codes below. Everything after this step is organised by code, so match yours and go directly to it.

If you need to be recording in the next two minutes, open Settings > Output, set Output Mode to Advanced, and change Video Encoder to Software (x264). That moves encoding to your CPU and works on every machine. Expect higher CPU load, and come back to the real fix afterwards.

Step 2: Driver Version and Clean Install

This is the fix for Function not implemented and INVALID_VERSION, and only those.

Both mean the driver and the NVENC SDK that OBS was built against do not agree. That happens when the driver is genuinely old, and it also happens when a routine driver update layered itself over a previous install and carried a corrupted encode profile forward. This is the reason "I already updated my drivers" and "I clean installed my drivers" are different statements with different outcomes.

1. Find your exact card model: press Windows + R, type dxdiag, press Enter, and read the Display tab.
2. Download the current driver for that model from nvidia.com directly. Not Windows Update, not a third-party driver utility, not GeForce Experience's in-place updater.
3. Run the installer and choose Custom (Advanced) instead of Express.
4. Tick Perform a clean installation.
5. Restart, then set Video Encoder back to NVIDIA NVENC H.264 in OBS.

Either the Game Ready or Studio branch works; Studio drivers are validated on a slower cadence and are the safer choice if you record for work.

If a clean install does not fix it, install a driver from a few releases back rather than the newest. NVIDIA does occasionally ship a branch with a broken encode path, and OBS forum threads for a given driver number will tell you within a day whether that is what you are hitting.

Step 3: NVENC Session Limits and VRAM

NV_ENC_ERR_OUT_OF_MEMORY is the code people most reliably misread. It usually does not mean your VRAM is full.

NVENC supports a limited number of concurrent encode sessions. Consumer GeForce cards have historically been capped at a small number — this limit has been raised over the years, and it is still a limit. Every application holding a session counts against it, and a session held by a crashed process is not released until that process is gone.

Things that hold an NVENC session, often without you realising:

• NVIDIA App / GeForce Experience Instant Replay or ShadowPlay. The most common single cause. It holds a session continuously while enabled, not only while recording.
• A second OBS instance, including one that crashed and left a process behind. Check Task Manager for obs64.exe and end any orphans.
• Discord's Go Live or screen share with hardware acceleration on.
• Browser hardware acceleration with video playing.
• Xbox Game Bar background recording.
• Any game using NVIDIA's in-game recording features.

Close them, then retry. If it now works, re-enable them one at a time to find which one you can live without.

If it genuinely is VRAM: running a game at 4K with high texture settings on an 8 GB card can leave too little for an encode buffer. Lower the game's texture quality, or lower the OBS Output (Scaled) Resolution under Settings > Video.

Step 4: OBS Is Talking to the Wrong GPU

UNSUPPORTED_DEVICE and no NVENC capable devices found both mean the same thing: OBS asked for hardware encoding from a chip that does not have NVENC. On laptops and most pre-built desktops, that chip is the integrated graphics on your CPU.

Windows decides which GPU each application uses, and it defaults new applications to the power-saving one.

Assign OBS to the NVIDIA GPU:

1. Press the Windows key, type Graphics settings, press Enter.
2. Under Custom options for apps, click Browse and select obs64.exe — normally C:\\Program Files\\obs-studio\\bin\\64bit\\obs64.exe.
3. Click Options, choose High performance, and Save.
4. Restart OBS.

Set the same preference in the NVIDIA Control Panel under Manage 3D settings > Program Settings: add OBS, and set Preferred graphics processor to High-performance NVIDIA processor.

Laptop specifics. On battery, many laptops disable the discrete GPU entirely to save power, and NVENC disappears with it. Plug in before recording and set the Windows power mode to Best performance. If your laptop has a MUX switch or an "Ultimate / Discrete GPU" mode in its vendor utility, enabling it removes this class of problem permanently.

Disabling the integrated GPU in Device Manager also works and is what a lot of guides recommend, but on a laptop it can cost you battery life and occasionally break display output. Try the graphics-preference route first.

Step 5: Settings NVENC Will Not Accept

NV_ENC_ERR_INVALID_PARAM means the encode configuration itself is invalid for your card.

• Resolution or frame rate above the card's encode limit. Older cards cannot encode 4K at 60 fps even when they can render it. Lower Output (Scaled) Resolution in Settings > Video and test.
• B-frames on a card that does not support them. In Settings > Output > Recording or Streaming, set Max B-frames to 0 and test.
• HEVC or AV1 selected on a card that does not have that encoder. AV1 encode requires a recent generation. Switch to NVIDIA NVENC H.264.
• A corrupted profile. If you migrated OBS from another machine or restored a backup, the encoder parameters may reference hardware you no longer have. In the menu bar: Profile > New, then Scene Collection > New, configure NVENC in the clean profile, and test. If it works there, rebuild rather than repair.

Symptom Router

The Distinction Worth Keeping

NVENC failing is not the same as OBS being slow. If OBS starts, records, and the result stutters or drops frames, the encoder initialised fine and you have a throughput problem instead — OBS Encoding Overloaded Error and OBS Encoder Lag: NVENC vs x264 cover that. This page is only about NVENC refusing to start at all.

Why You Are Choosing an Encoder in the First Place

OBS asks you to select an encoder because it is a broadcast tool, and broadcast work genuinely needs that choice: pinning a specific bitrate to a specific hardware path matters when you are running a multi-camera production with a hardware capture card.

It matters considerably less when you are recording a five-minute product walkthrough. The encoder dropdown is where the driver state, the session limit, the GPU assignment and the profile history all meet, and it will keep failing at that intersection.

Cubix Capture does not expose one. It resolves the capture path itself on Windows and Mac, and applies the zoom and cursor smoothing during capture rather than in an edit pass afterwards. The free Starter plan runs without an account, which is enough to compare the output to what you are producing now.

Keep Reading

• OBS "Starting the Output Failed": Fix for output failures that are not NVENC.
• OBS Encoder Lag: NVENC vs x264 once NVENC starts but cannot keep up.
• OBS High CPU Usage: How to Reduce It if you fell back to x264 and felt it.
• OBS Best Settings for Screen Recording for a known-good baseline configuration.