OBS Fails to Properly Distribute Encoder Load on M3 Ultra (macOS)

[ninbura]

Operating System Info

macOS 26

Other OS

No response

OBS Studio Version

32.1.2

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/u9mvWmyADlYSceFN

OBS Studio Crash Log URL

No response

Expected Behavior

A Mac Studio w/ M3 Ultra (which has 4x H264/H265 encoders) should be capable of encoding more than one instance of 4K60 at once via OBS.

Current Behavior

When you stream in 4K60 and then try to record in 4K60 via Apple's hardware HEVC/H264 encoders with two different sets of encoder options via the M3 Ultra you get encoder overload.

Steps to Reproduce

1. Set base canvas to 3840x2160 (no scaling)
2. Set Streaming & Recording encoders to Apple VT HEVC/H264 Hardware Encoder.
   • CBR for streaming
   • CRF for recording
3. Start streaming & recording.

Anything else we should know?

I am able to encode 2 instances of 4K60 in real-time on this same system via FFmpeg, utilizing the same hardware encoders. I'm hitting 2 live sources with this test, 2x Elgato 4K X capture cards connected to 4K60+ sources.

I'm wondering if OBS is passing some parameter to AVFoundation that prevents it from using more than one encoder?

[tt2468]

Please specify the exact FFmpeg command that you are using to obtain the encode throughput that you are referring to.

[ninbura]

ffmpeg -hide_banner \
  -f avfoundation \
  -probesize 9223372036854775807 \
  -framerate 60 -video_size 3840x2160 -pixel_format nv12 -thread_queue_size 1024 \
  -i "2:9" \
  -c:v hevc_videotoolbox -q:v 72 -realtime true -g 120 -tag:v hvc1 -pix_fmt nv12 \
  -c:a aac -b:a 320k -ar 48000 -ac 2 \
  -movflags +frag_keyframe+empty_moov \
  video-output/camera.mp4

ffmpeg -hide_banner \
  -f avfoundation \
  -probesize 9223372036854775807 \
  -framerate 60 -video_size 3840x2160 -pixel_format nv12 -thread_queue_size 1024 \
  -i "1:7" \
  -c:v hevc_videotoolbox -q:v 72 -realtime true -g 120 -tag:v hvc1 -pix_fmt nv12 \
  -c:a aac -b:a 320k -ar 48000 -ac 2 \
  -movflags +frag_keyframe+empty_moov \
  video-output/appletv.mp4

Here's a screenshot of me capturing via FFmpeg and OBS @ 4K60 simultaneously in realtime with no encoder overload:

./raw-card-3.zsh
Input #0, avfoundation, from '1:7':
  Duration: N/A, start: 18044.832967, bitrate: N/A
  Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12, 3840x2160, 60 fps, 60 tbr, 1000k tbn, start 18044.832967
  Stream #0:1: Audio: pcm_f32le, 48000 Hz, stereo, flt, 3072 kb/s, start 18044.839208
File 'video-output/appletv.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (hevc_videotoolbox))
  Stream #0:1 -> #0:1 (pcm_f32le (native) -> aac (native))
Press [q] to stop, [?] for help
Output #0, mp4, to 'video-output/appletv.mp4':
  Metadata:
    encoder         : Lavf62.12.102
  Stream #0:0: Video: hevc (hvc1 / 0x31637668), nv12(tv, progressive), 3840x2160, q=2-31, 60 fps, 15360 tbn
    Metadata:
      encoder         : Lavc62.28.102 hevc_videotoolbox
  Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 320 kb/s
    Metadata:
      encoder         : Lavc62.28.102 aac
[out#0/mp4 @ 0x719418e40] video:13868KiB audio:799KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.187634%
frame= 2316 fps= 60 q=-0.0 Lsize=   14694KiB time=00:00:40.61 bitrate=2963.7kbits/s speed=1.05x elapsed=0:00:38.72    
[aac @ 0x718875f80] Qavg: 65286.230
Exiting normally, received signal 2.

https://obsproject.com/logs/jnDXmyz8k0uw8Lls

[ninbura]

Here's an example of me capturing 1 stream of 3440x1440@60FPS, and 2x streams of 4K60 simultaneously via the hevc_videotoolbox encoder without "encoder overload":

There's an issue in FFmpeg right now where it's very difficult to interact with more than 1 card of the same model at the same time. The cards are constantly shuffling positions, and you can't select them via their names as they're identical. I had to patch the code to allow for selecting capture devices via their serial numbers. So the -video_device_serial & -audio_device_serial options are not currently in a production release of FFmpeg, FYI. My build was a bit rough when I first opened this issue, but now it's stable.

That all being said, the point stands, no encoder overload.

./ffmpeg -hide_banner \
  -f avfoundation \
  -probesize 100M \
  -framerate 60 -video_size 3440x1440 -pixel_format nv12 -thread_queue_size 1024 \
  -video_device_serial A7SNB504219J0R -audio_device_serial A7SNB504219J0R -i "" \
  -c:v hevc_videotoolbox -q:v 72 -realtime true -g 120 -tag:v hvc1 -pix_fmt nv12 \
  -c:a aac -b:a 320k -ar 48000 -ac 2 \
  -movflags +frag_keyframe+empty_moov \
  -y video-output/ultrawide.mp4

./ffmpeg -hide_banner \
  -f avfoundation \
  -probesize 100M \
  -framerate 60 -video_size 3840x2160 -pixel_format nv12 -thread_queue_size 1024 \
  -video_device_serial A7SNB50424UBQI -audio_device_serial A7SNB50424UBQI -i "" \
  -c:v hevc_videotoolbox -q:v 72 -realtime true -g 120 -tag:v hvc1 -pix_fmt nv12 \
  -c:a aac -b:a 320k -ar 48000 -ac 2 \
  -movflags +frag_keyframe+empty_moov \
  -y video-output/camera.mp4

./ffmpeg -hide_banner \
  -f avfoundation \
  -probesize 100M \
  -framerate 60 -video_size 3840x2160 -pixel_format nv12 -thread_queue_size 1024 \
  -video_device_serial A7SNB50423R73R -audio_device_serial A7SNB50423R73R -i "" \
  -c:v hevc_videotoolbox -q:v 72 -realtime true -g 120 -tag:v hvc1 -pix_fmt nv12 \
  -c:a aac -b:a 320k -ar 48000 -ac 2 \
  -movflags +frag_keyframe+empty_moov \
  -y video-output/appletv.mp4

However, in OBS Studio, if you try to stream & record with two HEVC/H264 hardware sessions you immediately get "encoder overload".

[ninbura]

You can easily run the same test on any Mac running OBS with a 4K canvas. Took me a second to understand why OBS hardcodes Virtual Camera's resolution to 1080p regardless of your canvas resolution. But if you simply omit the resolution parameter in the FFmpeg command it'll set the resolution properly based on the feed.

cd "${0:A:h}"
zmodload zsh/datetime
mkdir -p video-output/obs-test # <-------- Note that this will create this directory relative to your current directory.
strftime -s stamp '%Y-%m-%d_%H-%M-%S' $EPOCHSECONDS
frac=${EPOCHREALTIME##*.}
out="video-output/obs-test/obs_${stamp}.${frac[1,3]}_$$.mp4"

ffmpeg -hide_banner \
  -f avfoundation \
  -probesize 100M \
  -framerate 60 -pixel_format nv12 -thread_queue_size 1024 \
  -i "OBS Virtual Camera" \
  -c:v hevc_videotoolbox -q:v 72 -realtime true -g 120 -tag:v hvc1 -pix_fmt nv12 \
  -movflags +frag_keyframe+empty_moov \
  "$out"

Spin that up 3 times, and monitor the info line in FFmpeg:

Worth noting that I had my replay buffer running in OBS during this test as well, which means I had 4x instances of 4K60 encoding at once with no encoder warnings or obvious degradations in capture performance.