[kakra]

Note

This remains a draft/RFC and a reference for the design, not something to merge as a whole. The Xlib/XWayland implementation has received broad runtime testing, while native Wayland capture is still untested at runtime. I keep this branch rebased and will split individual layers out as separate PRs when you are interested, and rebase this branch after each merge until nothing is left.

Feedback on state ownership, application-input inhibition, and backend boundaries would be especially useful before any of this is prepared for integration.

Summary

This series gives the vkShade overlay an independent mouse-capture path instead of relying on the application's absolute cursor position.

It:

• models capture requests, application-input inhibition, and native acquisition as independent states;
• integrates a bounded virtual cursor from absolute, root-relative, or relative motion;
• suppresses application mouse input through optional SDL and Wine adapters;
• captures Xlib/XWayland input through private connections, with an XI2 physical-device fallback;
• captures native Wayland input through relative-pointer and pointer-constraints protocols;
• handles focus changes, asynchronous acquisition, surface replacement, and symmetric teardown.

How the series is organized

The commits are ordered in layers so that they can be picked apart. Each commit builds on its own.

1. Wayland client context (refactor(platform): isolate Wayland client context): a private event queue and shared display state for the Wayland services.
2. Backend-neutral foundation: MouseCaptureController (capture state), VirtualMouseCursor, and MouseInputInhibitorGroup. None of these files mention Wayland, Xlib, or XCB.
3. Application inhibition: SDL2/SDL3 and Wine adapters.
4. Reconciliation and handoff: reconciling overlay capture requests and waiting for a confirmed release of an application-owned pointer constraint.
5. Native Wayland capture.
6. Xlib block, last: XInput raw motion decoding, Xlib capture, and the XI2 physical-source fallback.

Motivation

Some Wine and Proton applications keep controlling, centering, clipping, or exclusively grabbing the pointer while the vkShade GUI is visible.

Cyberpunk 2077 can leave the overlay cursor near the center while mouse movement continues to affect the game. Other engines use different cursor-warp, Raw Input, SDL, or native window-system paths. Treating a successful pointer grab as the complete capture state therefore fails across games and display backends.

The overlay instead maintains its own virtual cursor, inhibits the application's input path where possible, and reconciles native capture independently.

Architecture

Capture controller

MouseCaptureController owns the overlay capture request and reconciles:

1. application-side input inhibition;
2. readiness for native pointer ownership;
3. native backend acquisition;
4. retryable, pending, active, suspended, and unavailable states.

Failed native acquisition remains retryable without repeating successful inhibition transitions.

Application inhibition

MouseInputInhibitorGroup composes optional SDL and Wine adapters. It restores only adapters that acquired state and does so in reverse order.

SDL2 and SDL3 are resolved as separate runtime ABIs. SDL3 window operations use its asynchronous main-thread dispatcher, with generation tracking preventing stale work during rapid overlay close and reopen transitions.

Virtual cursor

VirtualMouseCursor centralizes bounds handling, relative integration, known application-warp rejection, and resynchronization after unknown absolute discontinuities.

The native backends provide observations while cursor policy remains backend-independent.

Xlib and XWayland

The Xlib backend owns private event connections and native grab state. When another X client retains the core pointer grab, vkShade observes physical XI2 source-device events for relative motion, buttons, and wheel input.

Events are filtered where deviceid == sourceid, avoiding duplicate master and source events. Capture metadata is retained across surface replacement only while the application connection remains compatible.

Native Wayland

The Wayland backend tracks the Vulkan surface and binds relative-pointer and pointer-constraints protocols on a vkShade-owned event queue.

Application inhibition must confirm that an existing game constraint has been released before vkShade requests its own constraint. This avoids the fatal Wayland already_constrained protocol error.

While capture is active, relative motion drives the shared virtual cursor and the native cursor is hidden. Surface replacement, seat capability loss, explicit request flushing, and protocol teardown remain isolated from the application's event queue.

The input backend is released when its Vulkan surface (or, as a fallback, the last instance) is destroyed, while the application still owns its Wayland connection. Before this, the backend's destructor ran at process exit and destroyed proxies of a display connection that the application had already closed, which crashed at exit.

Scope

This draft covers overlay mouse capture on Xlib/XWayland and native Wayland.

Keyboard isolation remains a separate concern: keyboard events can still reach the application while the overlay is open. An XCB native-acquisition backend and further query caching remain possible follow-ups.

Verification

Build:

• release build
• every commit builds on its own

I checked the state handling, inhibitor composition, SDL ABI isolation, virtual cursor, and XI2 decoding with unit tests. They are not part of this PR.

Manual Xlib/XWayland testing:

• Euro Truck Simulator 2
• Cyberpunk 2077
• Hogwarts Legacy
• Star Wars Outlaws
• Elite Dangerous
• Forza Horizon 5
• focus loss, surface recreation, click-through prevention, and restore
• vertical and horizontal wheel input

Native Wayland testing:

• startup and teardown on a real VK_KHR_wayland_surface path (vkcube with the Khronos validation layer; this found and fixed the exit-time crash mentioned above)
• relative capture and application-input suppression
• focus loss, surface replacement, and teardown with capture active
• fractional desktop scaling scenario reported in bug: mouse offset when using desktop scaling #134

Closes #39
Closes #134

[kakra]

@ralgar As announced, here it is.

[ralgar]

Yep, this is quite a lot. XD

I'll build it and test behavior when I get a moment. Aside from that, I'll wait until I get the uniform UI plumbing finished (it's going well so it shouldn't be long), and then I can turn my full attention to this.

I assume the fix you said I might want to cherry pick is that first commit?

[kakra]

Yep, this is quite a lot. XD

I told you... :-D

But if you feel like it and if possible, we can break that down into smaller pieces maybe. The commit series is bisectable, so it should work by just picking one or two commits at a time (at least after the fixup commits are squashed).

I assume the fix you said I might want to cherry pick is that first commit?

Yes, it fixes Hogwarts Legecy which doesn't receive mouse clicks if vkShade is loaded, even if the GUI is not active:

• fix(input): observe Xlib wheel without core events

[kakra]

One small design question before I continue:

InputBackendXlib::initialize_virtual_cursor() and synchronize_cursor_to_pointer() currently repeat parts of the window geometry, root-coordinate translation, bounds, and pointer-query logic.

I deliberately left this untouched for now. Refactoring it only inside the capture commit would make the shared abstraction look capture-specific. It might be cleaner to prepare a small, behavior-neutral Xlib geometry/pointer-snapshot helper on top of main first, and then build the capture work on that interface.

Would you prefer such a preparatory patch as a separate small PR, leave the duplication in this draft until the capture architecture is settled, or use a different ownership boundary?

[kakra]

I've added a few fixups which insert comments into the code to make it easier to review.

[ralgar]

But if you feel like it and if possible, we can break that down into smaller pieces maybe. The commit series is bisectable, so it should work by just picking one or two commits at a time (at least after the fixup commits are squashed).

Would you prefer such a preparatory patch as a separate small PR, leave the duplication in this draft until the capture architecture is settled, or use a different ownership boundary?

I figured as much, that seems to be your usual workflow. I think for now let's just leave it all together so the changes can be viewed as one unit. As I work through it I would probably break it down into smaller pieces, though.

If you could squash those fixup commits that would be great btw, then I can step through them individually and still view the comments. Whenever you get a moment, no rush.

[kakra]

Updated the branch with explicit SDL2/SDL3 ABI-safe symbol resolution, synchronized SDL event-filter state, and native SDL3 relative-mouse inhibition. SDL3 window operations are dispatched asynchronously to its main thread, with generation tracking preventing stale work during rapid overlay state changes. I also autosquashed the existing fixups, reordered the commits semantically, rebased onto current main, and successfully smoke-tested the result with ETS2.

[ralgar]

Would you mind cherry-picking that first commit and rebasing on main when you get a moment? I would do it myself, but there are a couple of minor conflicts and I don't want to break your signature.

The GUI work will likely be completed by the end of this weekend, which will finally mark v0.1 and graduate the project to alpha status. So I'd like to get that fix merged as well.

I still think the whole Xlib backend needs an overhaul, and I'm experimenting with some stuff there on the side, but no concrete action will be taken until after v0.1 is released and any initial bugs are worked out.

[kakra]

Would you mind cherry-picking that first commit and rebasing on main when you get a moment?

Sure, will do.

I still think the whole Xlib backend needs an overhaul, and I'm experimenting with some stuff there on the side, but no concrete action will be taken until after v0.1 is released and any initial bugs are worked out.

Maybe it does. By researching the different sources of why mouse capture doesn't properly work in one or another game, finding that each may work different, I think vkShade might not only need a different Xlib backend per-se, but a generic input backend design with input inhibitors in mind. This PR does not exist as a request to merge as-is but rather as a research to find all the different aspects of mouse capture, and how it could be solved, discovering their input implementations.

If time comes, we should treat this PR as an issue report and a reference with code examples rather than something I'd want to merge as-is. It's also rather big and not something intended to review as a whole.

[kakra]

The branch has been rebased onto current main and the existing fixups have been squashed into their owning commits.

The main functional addition since the previous review is native Wayland mouse capture. It uses relative-pointer and pointer-constraints protocols on vkShade's isolated event queue and waits for confirmed release of an application-owned constraint before acquiring its own. The Xlib path also gained the XI2 physical-source fallback, SDL2/SDL3 ABI-safe inhibition, and stronger capture-state reconciliation.

The final branch builds successfully and passes all 12 tests. The Xlib/XWayland path retains its existing multi-game runtime coverage. Native Wayland runtime testing is still pending, particularly the fractional-scaling scenario from #134.

[kakra]

Updated and force-pushed. The branch is now ordered in layers (Wayland client context, backend-neutral foundation, application inhibition, reconciliation and handoff, Wayland capture, Xlib last), see the new section in the description.

• Fixups for bugs I had introduced within this branch are folded into their commits (SDL ABI binding and filter synchronization into the SDL/Wine commit, the cursor resync into the Xlib capture commit).
• Testing on native Wayland with vkcube and the validation layer found a crash at process exit: the input backend was destroyed from a static destructor after the application had already closed its Wayland connection. The backend is now released when its surface is destroyed. That fix is part of the Wayland capture commit.
• I moved the unit tests out of this branch. They stay in my local integration branch. I can add a test for a specific bug if you want one.

The branch is rebased onto current main (v0.1.0). The diff of the series is unchanged by that.