Skip to content

OpenFX: remove reachable panics across the C ABI, null-check host GPU pointers, declare render InstanceSafe - #61

Open
Mldphotohraphie wants to merge 4 commits into
gyroflow:mainfrom
Mldphotohraphie:upstream-pr
Open

OpenFX: remove reachable panics across the C ABI, null-check host GPU pointers, declare render InstanceSafe#61
Mldphotohraphie wants to merge 4 commits into
gyroflow:mainfrom
Mldphotohraphie:upstream-pr

Conversation

@Mldphotohraphie

Copy link
Copy Markdown

Context

DaVinci Resolve Studio 21.0.2 (macOS arm64, Apple M-series) crashed with SIGSEGV while using Gyroflow OFX v2.1.1 during Color page playback, right after an Edit -> Color page switch. Crash dump excerpt (from Resolve's ResolveDebug.txt):

3   Metal            -[_MTLCommandQueue commitCommandBuffer:wake:] + 268
5   IOGPU            -[IOGPUMetalCommandBuffer commit] + 228
6   AGXMetalG15X_M1  -[AGXG15XFamilyCommandBuffer commit] + 880
7   Gyroflow.ofx     OfxGetPlugin + 9231056
...
Signal Number = 11

While auditing the OpenFX plugin code around this crash, I found several robustness issues that this PR addresses. It does not claim to fix the root cause of the crash above (which likely involves the lifetime of the host's Metal command queue retained by the cached wgpu wrapper in gyroflow-core), but it removes several ways the plugin can take the host down and reduces concurrent access to the shared render state.

Changes

  1. Remove reachable panics/unwraps from render and param paths. The plugin is loaded across the OpenFX C ABI; main_entry has no catch_unwind, so any Rust panic unwinding out of an action is undefined behavior in the host process. Replaced reachable panic!/unwrap() in frame_from_timetype, get_bool_at_time, set_string and thread join with logged fallbacks or error returns.

  2. Null-check host GPU pointers before handing them to the backend. The OpenCL/Metal/CUDA render paths passed raw host pointers (command queue, source/output buffers) straight to the GPU backend, where a null would be UB (Retained::retain(...).unwrap() on the Metal side). The render action now fails cleanly instead.

  3. Avoid unwrap on missing fuscript executable.

  4. Declare render as InstanceSafe instead of FullySafe. The Render action takes &mut InstanceData and mutates per-instance state (stabilization manager cache, timeline size, parameters). FullySafe allows the host to call Render concurrently on the same instance, which aliases that &mut (UB) and races those mutations. InstanceSafe still allows concurrent rendering of different instances.

Notes

Two further issues were identified but need changes in gyroflow-core rather than this repo, so they are not part of this PR:

  • std::env::set_var (NO_OPENCL in disable_opencl) is called on the render path while gyroflow-core reads these variables from render threads; setenv/getenv racing is UB on Unix. A shared atomic flag would be safer.
  • The wgpu wrapper caches (and retains) the host's MTLCommandQueue across renders; if the host tears down its GPU pipeline (e.g. Resolve page switches), a later commit on that queue can crash. Happy to file a separate issue with the full crash dump if useful.

Tested: cargo check / cargo build --release clean on macOS arm64 (rustc 1.94.1); the resulting build loads and renders correctly in Resolve Studio 21.0.2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF

@CLAassistant

CLAassistant commented Jul 23, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Mldphotohraphie and others added 4 commits July 23, 2026 12:42
These code paths run inside OpenFX actions dispatched across the C ABI,
so a panic here would abort the host process (or worse, with unwinding).

- frame_from_timetype: log and fall back to frame 0 instead of
  `panic!("Shouldn't happen")`. It is called from the parameter
  get/set-at-time callbacks.
- Render / GetRegionOfDefinition: `get_bool_at_time(DontDrawOutside)`
  now falls back to `false` (the parameter default) instead of
  unwrapping.
- check_pending_file_info: ignore a failed `set_string` instead of
  unwrapping, matching how set_string failures are handled elsewhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF
In the Render action the plugin reads raw pointers from the host image
properties (Metal/OpenCL command queue and image buffers) and passes
them to gyroflow-core, which retains the command queue and commits
command buffers on it. A null pointer there is undefined behaviour.

The downstream Metal path only null-checks the command queue, never the
buffers, and the OpenCL/CUDA paths check nothing. Validate all of them
up front and return a clean FAILED (with a log line) instead of feeding
a null pointer into the GPU backend.

Also guard the wgpu list_devices worker thread join against a panic
instead of unwrapping it during the Describe action.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF
CurrentFileInfo::query unwrapped get_fuscript() inside the worker
thread. The LoadCurrent button that triggers it is only defined when
the executable is available, but a race (file removed after describe)
would panic the worker. Return early with a log line instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF
The Render action receives `&mut InstanceData` and mutates per-instance
state: it takes `file_path`, updates the stabilization-manager cache,
timeline size and several parameters. Advertising FullySafe tells the
host it may call Render concurrently on the *same* instance, which would
hand out two `&mut InstanceData` (aliasing, undefined behaviour) and race
those parameter updates.

InstanceSafe serialises Render per instance while still allowing distinct
instances to render in parallel. This also reduces concurrent commits on
the host's Metal command queue through a shared stabilization manager.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnShHBDs8UYxcRKSAaZzEF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants