Fixed crash on presenting 0 size display

[aleksandr-voitenko]

Description

Fixed a tricky race condition crash between the rendering thread and some other thread that is in charge of a display creation, with the following stack trace:

OpenAdapter12
NDXGI::CDevice::RotateResourceIdentities
CRotateBackBufferIdentities<T>::RotateBackBufferIdentities
CDXGISwapChain::PresentImplCore
CDXGISwapChain::PresentImpl
CDXGISwapChain::Present
device_present (d3d11-subsystem.cpp:2559)
gs_present (graphics.c:1964)
render_display (obs-display.c:289)
render_displays (obs-video.c:104)
obs_graphics_thread_loop (obs-video.c:1253)
_acrt_getptd

Motivation and Context

The crash always happens in the graphics thread during an attempt to render data, but, as a symptom, display creation (obs_display_create), setting position, and resizing are always present near the end of logs on another thread.

The problem is that display initialization comprises 3 distinct steps, and each of them is a separate call. While they happen very fast (matter of milliseconds) and mutexes are correctly locked, my research shows that it is not sufficient.

The root cause of the problem is that we create OBS displays with zero width and height, which can cause issues in certain cases due to how the underlying libraries operate.

Creating a swap chain with zero dimensions is allowed, DXGI documentation explicitly states that

If you specify the width, height, or both (Width and Height members of DXGI_SWAP_CHAIN_DESC1 that pDesc points to) of the swap chain as zero, the runtime obtains the size from the output window that the hWnd parameter specifies.

The real problem is that the window's client area itself is 0x0 (common during early window creation, minimization, or certain window states). I checked this with additional logs, and it is exactly how this works in our app. In this case DXGI will create buffers with 0x0 dimensions and likely with zero size in bytes, which is certainly a straight path to undefined behavior.

So...

• The swap chain creation itself succeeds
• The crash occurs later when trying to use zero-sized resources; in our case, during rendering.
• Some evidence shows D3D11 validation errors like "buffer size cannot be zero" and crashes when dimensions are zero or even negative. I found a similar case with crash from Mozilla developers: https://bugzilla.mozilla.org/show_bug.cgi?id=1089364
• There is evidence that this is an edge case and might be drriver/hardware-dependent. So far, I've seen this crash only on NVIDIA GPUs, no AMD or Intel.

Summary:
There are multiple ways to fix this. Most of them require redesigning something in OBS. 😄
I've chosen the easiest one, just add a guard for dimensions and not try to render 0x0 window as it is anyway meaningless.

And finally:
As the crash is rare and unreproducible, all of this is based on guesswork and some evidence. 🔎
Fingers crossed it helps. 🤞

How Has This Been Tested?

Manually, Windows only.

Types of changes

• Bug fix (non-breaking change which fixes an issue)