Skip to content

Multi-GPU (PRIME) support for outputs on secondary GPUs#178

Open
Tarkhubal wants to merge 17 commits into
malbiruk:mainfrom
Tarkhubal:feat/multi-gpu-prime
Open

Multi-GPU (PRIME) support for outputs on secondary GPUs#178
Tarkhubal wants to merge 17 commits into
malbiruk:mainfrom
Tarkhubal:feat/multi-gpu-prime

Conversation

@Tarkhubal

Copy link
Copy Markdown

Closes #91.

Outputs wired to a secondary GPU now work alongside the primary GPU's outputs — e.g. a hybrid laptop whose internal panel is on the iGPU while the HDMI port is muxed to the dGPU. Both screens active at once, hotplug included.

How it works

driftwm never splits rendering across GPUs: every frame is composited on the primary GPU (first usable GPU at startup). What varies per output is scanout — outputs on the primary GPU scan out the composited buffer directly; outputs on another GPU scan out a copy allocated on their own GBM device (implicit PRIME, handled by smithay's GpuManager/MultiRenderer).

Full architecture write-up in dev/docs/multi-gpu.md. The main pieces:

  • Renderer abstraction: the whole compose + capture pipeline is generic over R: DriftRenderer, instantiated at MultiGpuRenderer (udev) and GlesRenderer (winit). Client content stays generic so secondary-GPU DrmCompositors can still promote it to planes; Gles-only effects (shader background, shadow, border, blur) bridge to the underlying GlesFrame via GlesBridge / as_gles_frame().
  • Device model: udev_devices: HashMap<DrmNode, UdevDevice>, one entry per KMS node, each with its own VBlank source and per-CRTC DrmCompositor. All GPUs are opened at startup; UdevEvent::Added/Removed add and remove whole devices at runtime. Split-DRM systems (several KMS nodes sharing one render node) are handled.
  • Per-surface dmabuf feedback: each output sends a render feedback (primary render node) or a scanout feedback (scanout-flagged tranches targeting the output's KMS device, Linear-only across GPUs) depending on the surface's render-element state — so fullscreen clients can reallocate into directly scannable buffers on either GPU. Buffers are also imported at commit time (early_import) to keep the cross-GPU copy off the render path.
  • Bridged-damage rule (new caveat, documented in CAVEATS.md): smithay's MultiFrame only PRIME-copies regions drawn through its own methods. Anything drawn via as_gles_frame() must record its damage with record_bridged_damage() — without it, secondary outputs accumulate stale trails wherever only bridged elements repaint. This also fixed a latent bug on main: GbmFramebufferExporter::new(gbm, None.into()) silently vetoed direct scanout of client buffers; the exporter now gets the device's render node.
  • Primary GPU unplug: no promotion of a new primary (same as niri) — surviving outputs go dark, but the dmabuf global and per-surface feedbacks are torn down cleanly so nothing advertises a dead node.

Single-GPU and winit paths are unchanged in behavior (everything monomorphizes to the same code as before).

Testing

Tested on real hardware (Dell G16, hybrid Intel Raptor Lake iGPU / NVIDIA RTX 4070 Mobile with nvidia-open, HDMI muxed to the dGPU), release build, both screens active:

  • Internal panel (Intel, 2560x1600@240) + external monitor (NVIDIA, 1920x1080@60) simultaneously; window dragging, camera pan/zoom, video playback on both — no artifacts, no visible lag.
  • Dmabuf feedback verified end-to-end: 122 scanout formats on the Intel output, 20 (Linear-only) on the NVIDIA output; WAYLAND_DEBUG on the client shows the expected tranches.
  • HDMI cable unplug + replug: output torn down and recreated correctly.
  • VT-switch away and back: both devices resume.
  • cargo fmt --check, clippy (0 warnings), 533 tests pass.
  • Tested on mkv videos, mp4 videos... also tested on video games (PAYDAY 2 and The Colonists, not tested on "bigger" games) via Steam

Not covered: unplugging the primary GPU (not physically possible on this bench — the teardown path is code-reviewed against niri's but untested on hardware).

Tarkhubal and others added 16 commits June 14, 2026 06:23
Prerequisite for PRIME/multi-GPU output support (malbiruk#91). Render code currently
hardcodes GlesRenderer everywhere; multi-GPU needs it generic over both the
concrete GlesRenderer and smithay's MultiRenderer.

Enable smithay's renderer_multi feature, add MultiGpuRenderer/MultiGpuFrame
type aliases, and introduce DriftRenderer/AsGlesRenderer/AsGlesFrame traits
(mirroring niri's render_helpers::renderer) implemented for both renderer
flavors. Not consumed by the render path yet; behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- `OutputRenderElements` is now generated by the `drift_render_elements!` macro
  as `OutputRenderElements<R: DriftRenderer = GlesRenderer>`. The default type
  parameter keeps every existing call site resolving to `GlesRenderer`, so the
  single-GPU path is unchanged while the enum is now instantiable with
  `MultiGpuRenderer`.
- `RoundedCornerElement` is now `RoundedCornerElement<R>` holding
  `WaylandSurfaceRenderElement<R>`, with dual `RenderElement` impls for
  `GlesRenderer` and `MultiGpuRenderer` (the corner-clip shader override is set
  on the bridged `GlesFrame`, then the surface is drawn through the `MultiFrame`).
- Added `RenderElement<MultiGpuRenderer>` impls for `PixelSnapRescaleElement<E>`
  and `TileShaderElement` (bridging via `as_gles_frame()`).
- Gles-only effect variants are wrapped in `GlesBridge`: `Background`
  (background/shadow/border) and `Blur`.
- Scanout-candidate variants (`Window`, `Layer`, `Cursor`, `Decoration`, …) stay
  generic over `R` so the secondary-GPU compositor can still promote them to
  planes.
- Build, clippy, and `cargo fmt --check` clean; existing tests pass. No change to
  the live (winit / single-GPU) render path.
- Render elements are now generic over the renderer (`OutputRenderElements<R>`,
  defaulting to `GlesRenderer`), so the same pipeline works with the plain
  `GlesRenderer` and the cross-GPU `MultiGpuRenderer`. Gles-only effects
  (background/shadow/border/blur) bridge through `GlesBridge`; window, layer, and
  cursor content stays generic so it can still be promoted to scanout planes.
- The live compose path (`compose_frame` and its helpers — layers, cursor,
  shadows/borders, blur, error bar, output outlines) is now generic over
  `R: DriftRenderer`. Offscreen blur work runs on the primary GPU via
  `as_gles_renderer()`. No behaviour change on the existing single-GPU / winit
  path.
- Made the capture path (wlr-screencopy and ext-image-copy-capture for outputs
  and toplevels) generic over the renderer: elements render through `R`, while
  pixel readback (`copy_framebuffer` / `map_texture`) runs on the primary GPU via
  `as_gles_renderer()`. The off-screen canvas screenshot path stays on the
  concrete `GlesRenderer`. No behaviour change on the single-GPU / winit path.
- Introduced a multi-GPU renderer manager on the udev backend: `Backend::Udev`
  now holds a `GpuManager` (one GLES renderer per DRM render node) plus the
  primary render node, and the render loop drives the primary output through
  `gpu_manager.single_renderer(node)` (a `MultiRenderer`). One-off renderer work
  (shader compilation, dmabuf import, off-screen screenshot) goes through a new
  `Backend::with_renderer` accessor. No behaviour change for a single-GPU setup;
  this is the groundwork for scanning out to displays on a secondary GPU.
- Reworked the udev backend to hold its DRM devices in a map keyed by KMS node
  (`DriftWm::udev_devices`) instead of a single `udev_device` handle, and made
  the render loop iterate the devices. Only the primary GPU is populated today;
  this is the structural prerequisite for adding a secondary GPU's device on
  hotplug. No behaviour change for a single-GPU setup.
- Made the udev render loop correct across multiple DRM devices: pending DPMS
  transitions are routed to the device that owns each output, and the
  output-management head list is aggregated across every device before
  notifying clients (previously each device overwrote the others'). Dirty-
  marking and rendering iterate every device. No behaviour change with a single
  GPU; this is groundwork for driving a second GPU's outputs.
- Multi-GPU (PRIME) support on the udev backend: every usable GPU is opened at
  startup (the system primary renders; outputs on other GPUs scan out via an
  implicit PRIME copy), whole GPUs hot-plug and hot-unplug at runtime (eGPU
  docks), and connector hotplug is routed to the owning device. Also fixes
  direct scanout of client buffers, which the framebuffer exporter previously
  vetoed for all devices.
Client buffers are imported on the primary GPU at commit time so the
cross-GPU copy overlaps client work instead of the render path. Removing
the primary GPU now tears down the dmabuf global and per-surface
feedbacks cleanly (no primary promotion, matching niri). Documents the
multi-GPU architecture in dev/docs/multi-gpu.md, the bridged-damage rule
in CAVEATS, and the smithay multigpu APIs.

Closes malbiruk#91
Copilot AI review requested due to automatic review settings July 3, 2026 08:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds multi-GPU (implicit PRIME) support to the udev backend so outputs on secondary GPUs can scan out a primary-GPU composited frame, including hotplug and per-surface dmabuf feedback.

Changes:

  • Introduces a renderer abstraction (DriftRenderer) and multi-GPU render-element plumbing (drift_render_elements!, GlesBridge) so the render pipeline can run on both single- and multi-GPU backends.
  • Refactors the udev backend to manage multiple DRM devices, select a primary render GPU, and render each output with GpuManager’s single- vs cross-GPU MultiRenderer.
  • Adds per-surface dmabuf feedback routing and early-import on wl_surface commit to keep cross-GPU copy/import off the render hot path.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/state/mod.rs Replaces single udev device handle with a per-GPU device map.
src/state/init.rs Initializes udev_devices map for non-udev backends.
src/render/shaders.rs Makes shader element builders generic over DriftRenderer and bridges Gles-only effects.
src/render/renderer.rs Adds DriftRenderer + AsGlesRenderer/AsGlesFrame traits for generic rendering.
src/render/render_elements_macro.rs Adds macro to generate renderer-generic render-element enums with per-renderer impls.
src/render/mod.rs Generalizes compose pipeline over DriftRenderer and exports new helpers.
src/render/lifecycle.rs Adds per-surface dmabuf feedback sending after successful frames.
src/render/layers.rs Generalizes layer rendering over DriftRenderer.
src/render/error_bar.rs Generalizes error bar rendering over DriftRenderer.
src/render/elements.rs Replaces smithay render_elements! with renderer-generic enum + bridges for Gles-only effects.
src/render/cursor.rs Generalizes cursor element building over DriftRenderer.
src/render/capture.rs Generalizes capture/screencopy rendering over DriftRenderer and uses as_gles_renderer() for readback.
src/render/capture_background.rs Bridges background capture element through GlesBridge.
src/render/bridge.rs Adds bridged-damage recording + GlesBridge adapter for Gles-only elements in multi-GPU.
src/render/blur.rs Updates blur pipeline to use underlying GlesRenderer for GlesTexture work and bridges blur element.
src/render/background.rs Makes background element generation renderer-generic; bridges shader background.
src/main.rs Updates udev init call pattern (no longer returns a device handle).
src/ipc/mod.rs Uses Backend::with_renderer closure for screenshots.
src/handlers/mod.rs Uses Backend::with_renderer for dmabuf import; updates gamma control to scan udev_devices.
src/handlers/compositor.rs Adds Backend::early_import on wl_surface commit.
src/backend/winit.rs Switches one-off renderer usage to Backend::with_renderer.
src/backend/udev.rs Implements multi-GPU device model, GpuManager integration, per-surface dmabuf feedback, hot(un)plug.
src/backend/mod.rs Replaces direct renderer access with with_renderer and adds early_import.
dev/docs/smithay-api.md Documents smithay multi-GPU APIs and caveats used by the implementation.
dev/docs/multi-gpu.md Adds architecture write-up for multi-GPU PRIME support.
dev/docs/CAVEATS.md Documents bridged-damage rule for multi-GPU.
CHANGELOG.md Adds unreleased entry for multi-GPU support.
Cargo.toml Enables smithay renderer_multi feature.
Cargo.lock Adds new transitive dependency for smithay multi-GPU support.
Comments suppressed due to low confidence (1)

src/render/capture.rs:409

  • render_to_dmabuf uses renderer.bind(dmabuf) for client-provided dmabufs. In the cross-GPU case, MultiRenderer’s Bind<Dmabuf> may target the scanout GPU, but capture/screencopy dmabufs are typically allocated for the primary render node (per advertised dmabuf feedback). This mismatch can make dmabuf-based screencopy/capture fail specifically on secondary-GPU outputs.
    use smithay::backend::allocator::Buffer;
    use smithay::backend::renderer::damage::OutputDamageTracker;

    let clear = clear_color_for(dmabuf.format().code);
    let sync = match capture_state {
        Some(cs) => {
            let mut target = renderer.bind(dmabuf)?;
            let result = cs
                .damage_tracker
                .render_output(renderer, &mut target, cs.age, elements, clear)?
                .sync;
            cs.age += 1;
            result
        }
        None => {
            let mut target = renderer.bind(dmabuf)?;
            let mut damage_tracker = OutputDamageTracker::new(size, scale, transform);
            damage_tracker
                .render_output(renderer, &mut target, 0, elements, clear)?
                .sync
        }
    };

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/backend/mod.rs Outdated
Comment on lines +27 to +36
pub fn with_renderer<T>(&mut self, f: impl FnOnce(&mut GlesRenderer) -> T) -> T {
match self {
Backend::Winit(backend) => backend.renderer(),
Backend::Udev(renderer) => renderer.as_mut(),
Backend::Winit(backend) => f(backend.renderer()),
Backend::Udev(udev) => {
let mut renderer = udev
.gpu_manager
.single_renderer(&udev.primary_render_node)
.expect("primary GPU renderer unavailable");
f(renderer.as_gles_renderer())
}
Comment thread src/render/capture.rs Outdated
Comment on lines 291 to 332
@@ -303,9 +313,9 @@ fn render_to_offscreen(
}
cs.age += 1;

let target = renderer.bind(tex)?;
let mapping =
renderer.copy_framebuffer(&target, Rectangle::from_size(buffer_size), format)?;
let gles = renderer.as_gles_renderer();
let target = gles.bind(tex)?;
let mapping = gles.copy_framebuffer(&target, Rectangle::from_size(buffer_size), format)?;
Ok(mapping)
} else {
let mut texture: GlesTexture =
@@ -315,9 +325,9 @@ fn render_to_offscreen(
let mut damage_tracker = OutputDamageTracker::new(size, scale, transform);
let _ = damage_tracker.render_output(renderer, &mut target, 0, elements, clear)?;
}
let target = renderer.bind(&mut texture)?;
let mapping =
renderer.copy_framebuffer(&target, Rectangle::from_size(buffer_size), format)?;
let gles = renderer.as_gles_renderer();
let target = gles.bind(&mut texture)?;
let mapping = gles.copy_framebuffer(&target, Rectangle::from_size(buffer_size), format)?;
Ok(mapping)
}
Comment thread src/backend/udev.rs
Comment on lines 1946 to 1952
// Fulfill capture requests after main render
#[cfg(feature = "profile-with-tracy")]
let _captures_span = tracy_client::span!("udev::captures");
let renderer = backend.renderer();
crate::render::render_screencopy(data, renderer, output, &elements);

let renderer = backend.renderer();
crate::render::render_capture_frames(data, renderer, output, &elements);

let renderer = backend.renderer();
crate::render::render_toplevel_captures(data, renderer);
crate::render::render_screencopy(data, &mut renderer, output, &elements);
crate::render::render_capture_frames(data, &mut renderer, output, &elements);
crate::render::render_toplevel_captures(data, &mut renderer);
#[cfg(feature = "profile-with-tracy")]
Comment thread src/render/blur.rs
Comment on lines 308 to 318
let mut bg_tex = match state.render.blur_bg_fbo.take() {
Some((tex, cached_size)) if cached_size == output_size => tex,
_ => {
let Ok(t) =
Offscreen::<GlesTexture>::create_buffer(renderer, Fourcc::Abgr8888, out_buf_size)
else {
let Ok(t) = Offscreen::<GlesTexture>::create_buffer(
renderer.as_gles_renderer(),
Fourcc::Abgr8888,
out_buf_size,
) else {
return;
};
t
@malbiruk

malbiruk commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Thanks, this looks solid. The Copilot review raised a few things worth a look before I merge: all around cross-GPU correctness in the capture/blur paths, which your testing (video/games) wouldn't have exercised on the secondary output.

If you can confirm/fix (or explain why they're non-issues), I'll merge.

- The compositor no longer crashes when the primary GPU's renderer is
  unavailable (e.g. after unplugging the primary GPU): dmabuf imports fail
  cleanly and `driftwm msg screenshot` returns an error instead of aborting.
- Screencopy and image-copy-capture to shm buffers now read pixels back on the
  GPU that owns the offscreen texture, fixing captures of secondary-GPU
  outputs.
- Blur is disabled on outputs driven by a secondary GPU: its render passes
  cannot span two GPU contexts, so those windows render without blur instead
  of with a broken one.
@Tarkhubal

Copy link
Copy Markdown
Author

THe first 3 findings where true, the only one I did not truly fix is Copilot's finding #4 (blur on cross-GPU outputs) - i mitigated it (blur is skipped on those frames) instead of making it work :

Why cross-GPU blur cannot be implemented correctly with smithay's current API:

  1. The blur pipeline requires every texture to live in one GL context. The pipeline is: render the scene behind the window into bg_tex --> crop bg_tex into cache.texture --> run Kawase down/up passes between cache.texture and cache.scratch -> render the surface into bg_tex again for the alpha mask --> crop into cache.mask --> multiply. Each step samples a texture produced by the previous step, which is only possible if they all share a GL context
  2. The first step is pinned to the target (scanout) GPU. Rendering "the scene behind the window" means drawing OutputRenderElements, and those can only be drawn through R's frame (damage_tracker.render_output(renderer, ...)). That requires binding bg_tex through the MultiRenderer, and on a cross-GPU renderer Bind/Offscreen route to the target GPU (verified in smithay's multigpu/mod.rs: bind() goes to self.target.device.renderer_mut() when a target exists). So bg_tex is forced onto the secondary GPU's context.
  3. The last step is pinned to the primary (render) GPU. The blurred result is composited into the frame as a render element, and MultiFrame draws happen on the render GPU (the PRIME copy to the target happens later, at finish()). So the final blurred texture must live in the primary GPU's context.
  4. Steps 2–3 have nowhere valid to run. The intermediate passes use raw GlesRenderer calls (render_texture_from_to, custom shader programs). as_gles_renderer() only exposes the render GPU's GlesRenderer (MultiRenderer::as_mut() returns self.render.renderer_mut()), and smithay exposes no accessor for the target GPU's GlesRenderer. Even if it did, you'd then hold a target-GPU blurred texture that step 3 can't composite (wrong context for the final draw), and there is no public texture-transfer primitive between the two contexts other than the internal PRIME copy inside MultiFrame::finish()

So the input of the pipeline is only reachable on the secondary GPU, the output is only usable on the primary GPU, and the middle can only run on the primary - the data flow is structurally broken, not just a matter of moving a create_buffer call to the right renderer (which is all that was needed for the capture fix, where read-back via ExportMem routes to the owning GPU)

A real fix would require one of: (a) smithay exposing the target device's renderer plus a texture import/blit path between the two contexts, (b) rendering the "scene behind the window" a second time through a primary-only single_renderer - but the elements are typed against the frame's renderer instance and the GpuManager is already mutably borrowed for the frame, or (c) restructuring blur to source from the render-GPU shadow buffer that MultiFrame uses internally, which smithay doesn't expose either. All three are upstream-shaped changes, not something to hack into this branch

Given that, the honest behavior today is graceful degradation: windows on secondary-GPU outputs render without blur (same as blur_radius = 0), with a one-time log line, documented as a known limitation in dev/docs/multi-gpu.md. Copilot's diagnosis of the bug was correct, its implied remedy ("allocate on the right GPU") is not achievable for this pipeline

(also checked by Claude Fable 5, it confirmed)

@samukhin

samukhin commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

And how do niri, cosmic-comp, and halley handle this?

@Tarkhubal

Copy link
Copy Markdown
Author

That's a great question honestly, after a (long) analysis that's what i have : all three are smithay-based, and they represent three genuinely different multi-GPU strategies

niri - render on primary, PRIME-copy per frame (same architecture as driftwm).
FOr every output, niri builds one MultiRenderer per frame: gpu_manager.renderer(&primary_render_node, &output_render_node, format) (src/backend/tty.rs:1878). Scene elements are rendered generically through that renderer, and smithay's MultiFrame::finish() does the implicit PRIME copy of the damaged regions to the scanout GPU. Direct scanout stays enabled (FrameFlags::ALLOW_PRIMARY_PLANE_SCANOUT), and clients get dmabuf feedback so fullscreen apps can hit planes on either GPU. All one-off GPU work (dmabuf import, screenshots, screencast, screencopy) goes through with_primary_renderer, which is a single_renderer(primary) and returns Option - captures are re-rendered on the primary GPU rather than reusing the frame's cross-GPU renderer, which keeps offscreen render + readback in a single GL context. niri has no blur, so it never faces multi-context effect pipelines

cosmic-comp - per-output GPU picking, cross-GPU only as a fallback.
cosmic doesn'(t have a fixed "primry render GPU". For each output it runs render_node_for_output() (src/backend/kms/surface/mod.rs:1484): if the buffers of the windows actually visible on that output were allocated on the output's own GPU (or there are no client buffers), it renders the whole output with single_renderer(target) - zero cross-GPU traffic. Only when the visible content lives on another GPU does it build a cross-GPU renderer(render, target). Its offscreen work (postprocess/screen-filter passes, screencopy blits) goes either through a single_renderer(target) or through the MultiRenderer's own Offscreen/Bind/blit/ExportMem - which all route consistently to the target device - so it never mixes two GL contexts on one texture. This is the most bandwidth-optimal design (a dGPU-only workload never crosses the PCie bus), at te cost of significant complexity: renderers can change from frame to frame, and every effect/capture path has to be written against "whichever device owns this frame"

halley (if you're talking about https://github.com/saltnpepper97/halley) -- flatten everything on the primary GPU, ship one texture across.
halley composes every output's frame into an offscreen GlesTexture on the primary GPU using the raw GlesRenderer (composed_frame_texture_for_output + a custom draw pass, crates/halley-wl/src/backend/tty/drm.rs:~1600). Presentation is then trivial: for a primary-GPU otput the finished texture is drawn into the swapchain as a single TextureRenderElement; for a secondary-GPU oryGpuTextureElement and drawn throughgpu_manager.renderer(primary, target, format) (drm.rs:1762), so smithay's PRIME copy transfers one already-composited full frame. This completely sidesteps the split-context problem: blur and all effects run in the primary GLn scene, and the cross-GPU boundary only ever sees a flat texture. THe trade-offs are real, though: an extra full-screen composite pass for every output (including primary-GPU ones), FrameFlags::empty() on the composed path - so no plane/direct scanout of client buffers (halley bolts on a st path for fullscreen games), and per-window damage tracking atthe KMS level is lost since the compositor only ever presents one big texture

My PR sits on the niri model, and for the same reasons: it preserves smithay's per-element damage tracking, keeps direct scanout available on both GPUs (battery, latency, fullscreen video), and only PRIME-copies damaged regions instead of full frames. The one thing halley's model buys that my PR can't do today is effects on secondary-GPU outputs (their blur works there, here it's disabld), because they pay for it structurally on every output, every frame. If cross-GPU blur ever becomes a priority, the interesting middle ground is a hybrid : keep the current pipeline for primary-GPU outputs, and fall back to halley-style "compose to a primary-GPU texture, present the flat texture cross-GPU" only for secondary-GPU outputs. That would restore blur there at the cost of direct scanout on those outputs only - a reasonable trade, since cross-GPU direct scanout is already format-restricted (Linear-only tranches) in practice. cosmic's GPU-picking is the wrong fit for driftwm's design, a shared infinite canvas means aany window can appear on any output at any moment, so "render this output on its own GPU" would force constant buffer migration decisions that cosmic's workspace model mostly avoids

@malbiruk malbiruk left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff plus a local build. This is strong work — the renderer abstraction, bridged-damage handling, and per-surface dmabuf feedback are careful and faithful to niri's proven approach, and cargo build, clippy, and the full test suite (542 tests) are green on my end.

Two things I'd like resolved before merge, then this is good to go — both left as inline comments:

  1. A cross-device correctness bug in the frame-accounting maps (on src/backend/udev.rs, with a ready patch).
  2. The blur limitation is documented as a structural impossibility, but it isn't one (on dev/docs/multi-gpu.md).

Cross-GPU blur itself is fine to leave off for now — I've opened a separate tracking issue with the niri blueprint, so there's no need to expand this PR's scope. Thanks for the thorough hardware testing and the excellent architecture write-up.

Comment thread src/backend/udev.rs
// An armed estimated-VBlank timer counts as waiting, like frames_pending:
// re-rendering before either resolves spins render_frame past refresh rate.
if data.redraws_needed.contains(&surface.output)
&& !data.frames_pending.contains(&crtc)

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-device bug: frames_pending and estimated_vblank_timers are keyed by crtc::Handle, but that's a per-DRM-device kernel object ID — two GPUs routinely allocate overlapping CRTC handles. These maps are global on DriftWm and every device's VBlank closure mutates them by bare crtc, so on a multi-GPU + multi-output setup (the exact case this PR enables) device A's CRTC-H and device B's CRTC-H collide:

  • Starvation (this line): while A has a flip in flight on H, B's render is gated off even though B has nothing pending.
  • Render-on-top-of-in-flight-flip (UB): A's VBlank does frames_pending.remove(H) while B's flip on H is still unacked → B gets re-rendered mid-flip.
  • Timer clobber: B can't arm its estimated-vblank timer because A holds key H; A's removal drops B's entry.

It's single-GPU-invisible (one device → no collision), which is why the tests don't catch it, and it's enumeration-dependent so it won't reproduce on every box — but it silently breaks the configuration this PR exists for.

Fix: key both maps by Output (globally unique), matching redraws_needed right above them — which is also what niri does (output_state: HashMap<Output, OutputState>). Patch below is build- + clippy-clean with all 542 tests passing (keys both maps by Output at every site, drops the now-unused crtc params from render_frame/queue_estimated_vblank_timer, and moves the hotplug-disconnect cleanup inside the if let Some(surface) so it can key by the surface's output). Feel free to apply as-is or adapt:

diff --git a/src/backend/udev.rs b/src/backend/udev.rs
index 4f0d2fb..019df88 100644
--- a/src/backend/udev.rs
+++ b/src/backend/udev.rs
@@ -334,7 +334,7 @@ pub(crate) fn render_if_needed(data: &mut DriftWm) {
         for (output, on) in &pending {
             for device in &devices {
                 let mut dev = device.0.borrow_mut();
-                let Some((&crtc, surface)) =
+                let Some((_, surface)) =
                     dev.surfaces.iter_mut().find(|(_, s)| s.output == *output)
                 else {
                     continue;
@@ -349,8 +349,8 @@ pub(crate) fn render_if_needed(data: &mut DriftWm) {
                         );
                     }
                     data.redraws_needed.remove(output);
-                    data.frames_pending.remove(&crtc);
-                    if let Some(token) = data.estimated_vblank_timers.remove(&crtc) {
+                    data.frames_pending.remove(output);
+                    if let Some(token) = data.estimated_vblank_timers.remove(output) {
                         data.loop_handle.remove(token);
                     }
                 }
@@ -454,7 +454,7 @@ pub(crate) fn render_if_needed(data: &mut DriftWm) {
             continue;
         }
         let render_node = dev.render_node;
-        for (&crtc, surface) in dev.surfaces.iter_mut() {
+        for (_, surface) in dev.surfaces.iter_mut() {
             if data.dpms_off_outputs.contains(&surface.output) {
                 data.redraws_needed.remove(&surface.output);
                 continue;
@@ -462,10 +462,10 @@ pub(crate) fn render_if_needed(data: &mut DriftWm) {
             // An armed estimated-VBlank timer counts as waiting, like frames_pending:
             // re-rendering before either resolves spins render_frame past refresh rate.
             if data.redraws_needed.contains(&surface.output)
-                && !data.frames_pending.contains(&crtc)
-                && !data.estimated_vblank_timers.contains_key(&crtc)
+                && !data.frames_pending.contains(&surface.output)
+                && !data.estimated_vblank_timers.contains_key(&surface.output)
             {
-                render_frame(data, surface, crtc, render_node);
+                render_frame(data, surface, render_node);
             }
         }
     }
@@ -685,7 +685,7 @@ pub fn init_udev(
                                     );
                                 }
                             }
-                            render_frame(data, surface, crtc, render_node);
+                            render_frame(data, surface, render_node);
                         }
                     }
                 }
@@ -942,13 +942,13 @@ fn attach_gpu(data: &mut DriftWm, gpu: OpenedGpu) {
                             Ok(None) => {}
                             Err(e) => tracing::warn!("frame_submitted error: {e:?}"),
                         }
-                        data.frames_pending.remove(&crtc);
+                        data.frames_pending.remove(&surface.output);
                         // Real VBlank beat any estimated-VBlank timer we might have armed.
-                        if let Some(token) = data.estimated_vblank_timers.remove(&crtc) {
+                        if let Some(token) = data.estimated_vblank_timers.remove(&surface.output) {
                             data.loop_handle.remove(token);
                         }
                         if data.redraws_needed.contains(&surface.output) {
-                            render_frame(data, surface, crtc, render_node);
+                            render_frame(data, surface, render_node);
                         }
                     }
                     DrmEvent::Error(err) => {
@@ -1046,7 +1046,7 @@ fn scan_device_connectors(data: &mut DriftWm, device: &UdevDevice) {
                             &mut data.foreign_toplevel_state,
                             &surface.output,
                         );
-                        render_frame(data, surface, crtc, render_node);
+                        render_frame(data, surface, render_node);
                     }
                 }
                 DrmScanEvent::Connected {
@@ -1064,6 +1064,7 @@ fn scan_device_connectors(data: &mut DriftWm, device: &UdevDevice) {
                 } => {
                     tracing::info!("Hotplug: CRTC {crtc:?} disconnected");
                     if let Some(surface) = surfaces.remove(&crtc) {
+                        let output = surface.output.clone();
                         // "Last output" spans all devices: another GPU's
                         // monitor keeps the canvas alive.
                         let is_last = surfaces.is_empty()
@@ -1073,10 +1074,10 @@ fn scan_device_connectors(data: &mut DriftWm, device: &UdevDevice) {
                                 .filter(|d| !Rc::ptr_eq(&d.0, &device.0))
                                 .all(|d| d.0.borrow().surfaces.is_empty());
                         teardown_output(data, surface, is_last);
-                    }
-                    data.frames_pending.remove(&crtc);
-                    if let Some(token) = data.estimated_vblank_timers.remove(&crtc) {
-                        data.loop_handle.remove(token);
+                        data.frames_pending.remove(&output);
+                        if let Some(token) = data.estimated_vblank_timers.remove(&output) {
+                            data.loop_handle.remove(token);
+                        }
                     }
                 }
                 _ => {}
@@ -1131,9 +1132,9 @@ fn gpu_removed(data: &mut DriftWm, node: DrmNode) {
         .map(|d| d.0.borrow().surfaces.len())
         .sum();
     let count = surfaces.len();
-    for (i, (crtc, surface)) in surfaces.into_iter().enumerate() {
-        data.frames_pending.remove(&crtc);
-        if let Some(t) = data.estimated_vblank_timers.remove(&crtc) {
+    for (i, (_, surface)) in surfaces.into_iter().enumerate() {
+        data.frames_pending.remove(&surface.output);
+        if let Some(t) = data.estimated_vblank_timers.remove(&surface.output) {
             data.loop_handle.remove(t);
         }
         teardown_output(data, surface, surviving == 0 && i + 1 == count);
@@ -1739,7 +1740,6 @@ fn teardown_output(data: &mut DriftWm, surface: SurfaceData, is_last: bool) {
 fn render_frame(
     data: &mut DriftWm,
     surface: &mut SurfaceData,
-    crtc: crtc::Handle,
     render_node: DrmNode,
 ) {
     #[cfg(feature = "profile-with-tracy")]
@@ -1832,7 +1832,7 @@ fn render_frame(
         Err(e) => {
             tracing::warn!("Failed to acquire renderer for {render_node}: {e:?}");
             data.backend = Some(backend);
-            queue_estimated_vblank_timer(data, output, crtc);
+            queue_estimated_vblank_timer(data, output);
             return;
         }
     };
@@ -1927,23 +1927,23 @@ fn render_frame(
             };
             match queue_result {
                 Ok(()) => {
-                    data.frames_pending.insert(crtc);
+                    data.frames_pending.insert(output.clone());
                 }
                 Err(FrameError::EmptyFrame) => {
                     // No page flip - no real VBlank to wake us. Always arm the
                     // estimated timer so the render gate paces re-renders to the refresh
                     // period; otherwise a dirty-but-unchanged output spins render_frame.
-                    queue_estimated_vblank_timer(data, output, crtc);
+                    queue_estimated_vblank_timer(data, output);
                 }
                 Err(e) => {
                     tracing::warn!("Failed to queue frame: {e:?}");
-                    queue_estimated_vblank_timer(data, output, crtc);
+                    queue_estimated_vblank_timer(data, output);
                 }
             }
         }
         Err(e) => {
             tracing::warn!("Render frame error: {e:?}");
-            queue_estimated_vblank_timer(data, output, crtc);
+            queue_estimated_vblank_timer(data, output);
         }
     }
 
@@ -2029,9 +2029,9 @@ fn deliver_presentation(
 }
 
 /// Wake the VBlank-driven loop at ~one refresh period when queue_frame returned
-/// EmptyFrame, so ongoing animations keep ticking. Idempotent per CRTC.
-fn queue_estimated_vblank_timer(data: &mut DriftWm, output: &Output, crtc: crtc::Handle) {
-    if data.estimated_vblank_timers.contains_key(&crtc) {
+/// EmptyFrame, so ongoing animations keep ticking. Idempotent per output.
+fn queue_estimated_vblank_timer(data: &mut DriftWm, output: &Output) {
+    if data.estimated_vblank_timers.contains_key(output) {
         return;
     }
     // Clamp refresh mHz before the cast: negative i32 would wrap to a huge u64 and
@@ -2043,14 +2043,15 @@ fn queue_estimated_vblank_timer(data: &mut DriftWm, output: &Output, crtc: crtc:
         .unwrap_or_else(|| Duration::from_micros(16_667));
 
     let timer = Timer::from_duration(duration);
+    let timer_output = output.clone();
     match data
         .loop_handle
         .insert_source(timer, move |_, _, data: &mut DriftWm| {
-            data.estimated_vblank_timers.remove(&crtc);
+            data.estimated_vblank_timers.remove(&timer_output);
             TimeoutAction::Drop
         }) {
         Ok(tok) => {
-            data.estimated_vblank_timers.insert(crtc, tok);
+            data.estimated_vblank_timers.insert(output.clone(), tok);
         }
         Err(e) => tracing::warn!("Failed to insert estimated VBlank timer: {e:?}"),
     }
@@ -2074,15 +2075,15 @@ fn apply_pending_mode_changes(
 
     let mut unclaimed = HashMap::new();
     for (name, mut pm) in pending {
-        let Some((crtc, surface)) = surfaces.iter_mut().find(|(_, s)| s.output.name() == name)
+        let Some((_, surface)) = surfaces.iter_mut().find(|(_, s)| s.output.name() == name)
         else {
             unclaimed.insert(name, pm);
             continue;
         };
 
-        // Defer if a page flip is in flight on this CRTC — modesetting on top
+        // Defer if a page flip is in flight on this output — modesetting on top
         // of pending frames is undefined behavior.
-        if data.frames_pending.contains(crtc) {
+        if data.frames_pending.contains(&surface.output) {
             if pm.retry_count >= MAX_RETRIES {
                 tracing::error!(
                     "Mode change for '{name}' dropped after {MAX_RETRIES} deferrals (frames stuck pending)"
diff --git a/src/state/mod.rs b/src/state/mod.rs
index d51b4db..46bf869 100644
--- a/src/state/mod.rs
+++ b/src/state/mod.rs
@@ -73,7 +73,6 @@ use smithay::backend::session::libseat::LibSeatSession;
 use smithay::wayland::seat::WaylandFocus;
 
 use smithay::reexports::calloop::RegistrationToken;
-use smithay::reexports::drm::control::crtc;
 
 use crate::backend::Backend;
 use crate::input::gestures::GestureState;
@@ -552,10 +551,12 @@ pub struct DriftWm {
     /// Outputs whose CRTC is currently active. Universe for [`Self::mark_all_dirty`].
     pub active_outputs: HashSet<Output>,
     pub redraws_needed: HashSet<Output>,
-    pub frames_pending: HashSet<crtc::Handle>,
+    /// Outputs with a page flip in flight. Keyed by `Output`, not
+    /// `crtc::Handle`: CRTC ids are per-DRM-device and collide across GPUs.
+    pub frames_pending: HashSet<Output>,
     /// One-shot timers armed when queue_frame returned EmptyFrame so the loop
     /// still wakes at ~refresh rate to advance animations (e.g. xcursor frames).
-    pub estimated_vblank_timers: HashMap<crtc::Handle, RegistrationToken>,
+    pub estimated_vblank_timers: HashMap<Output, RegistrationToken>,
 
     pub config_file_mtime: Option<std::time::SystemTime>,
 

Comment thread dev/docs/multi-gpu.md
framebuffer/mapping. Never read an `R`-bound texture back through
`as_gles_renderer()`. The dmabuf capture path (`render_to_dmabuf`) has no
readback and is naturally cross-GPU-safe.
- **Blur is skipped on cross-GPU frames** (`frame_is_cross_gpu`, set per frame

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This frames cross-GPU blur as structurally impossible — "smithay exposes no handle to the target GPU's GlesRenderer, so there is currently no correct way to run it." That's a misdiagnosis: you never blur on the target (scanout) GPU. niri runs effects on the primary render GlesRenderer (via MultiRenderer::as_mut(), which you already expose as as_gles_renderer()), wraps the result as a primary-GPU texture element, and lets MultiFrame::finish() PRIME-copy the composited frame — no target-GPU renderer handle needed.

And this PR is already ~90% there: the blur passes already run on the primary via renderer.as_gles_renderer(), and the result is inserted via GlesBridge(blur_elem) drawing through as_gles_frame() + record_bridged_damage — the same shape as niri's PrimaryGpuTextureRenderElement. Only the blur source offscreen (behind-content → bg_tex) still binds on the target GPU, which is why frame_is_cross_gpu gates it off.

Could you reword this bullet (and the matching blur.rs comment + the CHANGELOG line) from "impossible / no correct way" to "not yet implemented"? The mechanism and the remaining work are captured in the follow-up issue. Suggested phrasing:

Blur is not yet implemented on cross-GPU frames (frame_is_cross_gpu, set per frame by the udev render loop) — a wiring gap, not a structural limit. The blur passes and the blurred-texture element already run on the primary GlesRenderer and composite via as_gles_frame(); only the blur source offscreen still binds on the target GPU. Routing that onto the primary GlesRenderer too (compute the effect source on the primary, PRIME-copy the finished frame) would enable it. Until then, windows on secondary-GPU outputs render without blur.

@malbiruk

malbiruk commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Tracking issue for the cross-GPU blur follow-up (the dev/docs/multi-gpu.md reword references it): #190.

@malbiruk
malbiruk force-pushed the main branch 2 times, most recently from 5663fc2 to dea3d12 Compare July 10, 2026 11:45
@maciejewiczow

maciejewiczow commented Jul 17, 2026

Copy link
Copy Markdown

I just tested binary built from this PR on my machine and I noticed some issues:

  1. At the first start driftwm used only my Nividia GPU, ignoring the Intel GPU, so the laptop builtin screen wasn't being utilized. The performance was nice though and otherwise everything seemed in order.
  2. At the second boot, it used both GPUs, but there performance was much worse, with flickering artifacts appearing on inactive windows. I tried to record the artifacts, but OBS couldn't capture the screen for some reason.
  3. It randomly refuses to start, either when switching from another dekstop env (KDE Plasma) or after a fresh boot. Not sure if this is related to the changes here, but it happens for me. There is this in the logs after a failed run:
Logs
systemd[2101]: Starting driftwm.service - driftwm, a trackpad-first infinite canvas Wayland compositor...
driftwm[2719]: 2026-07-16T13:59:41.114784Z  INFO driftwm::config: Loaded config from /home/mafciej/.config/driftwm/config.toml
driftwm[2719]: 2026-07-16T13:59:41.115153Z  INFO input_seat{name="seat-0"}:add_keyboard_with_context_flags{xkb_config=XkbConfig { rules: "", model: "", layout: "pl,us", varian>
driftwm[2719]: 2026-07-16T13:59:41.117888Z  INFO input_seat{name="seat-0"}:add_keyboard_with_context_flags{xkb_config=XkbConfig { rules: "", model: "", layout: "pl,us", varian>
driftwm[2719]: 2026-07-16T13:59:41.124218Z  INFO driftwm::backend::udev: Session created on seat: seat0
driftwm[2719]: 2026-07-16T13:59:41.124232Z  INFO driftwm::backend::udev: Backend config: wait_for_frame_completion=false, disable_direct_scanout=false, disable_hardware_cursor>
driftwm[2719]: 2026-07-16T13:59:41.129019Z  INFO driftwm::backend::udev: System primary GPU: /dev/dri/card2
driftwm[2719]: 2026-07-16T13:59:41.129039Z  INFO driftwm::backend::udev: GPU candidates: ["/dev/dri/card2", "/dev/dri/card1"]
driftwm[2719]: 2026-07-16T13:59:41.145205Z  WARN smithay::backend::drm::device::fd: Unable to become drm master, assuming unprivileged mode
driftwm[2719]: 2026-07-16T13:59:41.145215Z  INFO smithay::backend::drm::device: DrmDevice initializing
driftwm[2719]: 2026-07-16T13:59:41.145755Z ERROR drm_atomic: smithay::backend::drm::device::atomic: Failed to restore previous state. Error: Permission denied (os error 13)
driftwm[2719]: 2026-07-16T13:59:41.145777Z  WARN driftwm::backend::udev: /dev/dri/card2: failed to create DRM device (DRM access error: Failed to disable connectors on device >
driftwm[2719]: 2026-07-16T13:59:41.145793Z  INFO smithay::backend::drm::device::fd: Dropping device: Some("/dev/dri/card2")
driftwm[2719]: 2026-07-16T13:59:41.159120Z  WARN smithay::backend::drm::device::fd: Unable to become drm master, assuming unprivileged mode
driftwm[2719]: 2026-07-16T13:59:41.159128Z  INFO smithay::backend::drm::device: DrmDevice initializing
driftwm[2719]: 2026-07-16T13:59:41.159897Z ERROR drm_atomic: smithay::backend::drm::device::atomic: Failed to restore previous state. Error: Permission denied (os error 13)
driftwm[2719]: 2026-07-16T13:59:41.159912Z  WARN driftwm::backend::udev: /dev/dri/card1: failed to create DRM device (DRM access error: Failed to disable connectors on device >
driftwm[2719]: 2026-07-16T13:59:41.159921Z  INFO smithay::backend::drm::device::fd: Dropping device: Some("/dev/dri/card1")
driftwm[2719]: Error: "No usable GPU found (are you running from a TTY?)"
systemd[2101]: driftwm.service: Main process exited, code=exited, status=1/FAILURE
systemd[2101]: driftwm.service: Failed with result 'exit-code'.
systemd[2101]: Failed to start driftwm.service - driftwm, a trackpad-first infinite canvas Wayland compositor.

My hardware is Lenovo Legion y540 with NVIDIA GeForce GTX 1050 Ti (Driver Version: 580.159.03) and I am running Linux Mint on it

@maciejewiczow

Copy link
Copy Markdown

The artifacts are especially bad when multiple outputs overlap on the canvas.
I have my 2 external monitors connected over a Thunderbolt docking station, idk if that matters

@Tarkhubal

Copy link
Copy Markdown
Author

This is a separate issue from the multi-GPU work, it's a DRM master problem, not a rendering bug. The chain is:
Unable to become drm master --> Permission denied (os error 13) on the atomic reset --> both cards dropped --> No usable GPU found.
driftwm can't take DRM master, so it can't drive KMS at all. logind only hands master to the active session on the seat, and the "are you running from a TTY?" hint is the key: if it's launched as a systemd --user service (looks like it here systemd[2101]), that unit generally isn't attached to the active graphical session, so it never gets master. The fact that it's intermittent and worse right after leaving KDE points to a race where the previous session (KDE/SDDM) hasn't fully released the seat yet.
Quick check: on a failed run, what does loginctl session-status show for the session launching driftwm, is it "Active: yes" and attached to seat0? And is it started from a TTY / DM session entry, or via a systemd user service? That'll confirm whether it's a non-active session never getting master vs. a release race with the DM.
(sudo would "fix" it by forcing master as root, but that bypasses logind and will clash with the running session - not the right path.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add multi-GPU (PRIME) support for outputs on secondary GPUs

5 participants