Shared Scene Misalignment

[djmonte94]

I was wondering if you know of a Meta engineer that has worked on the world lock feature of the Meta SDK. We are able to share scenes across headsets, however we are running into a few alignment bugs. My team and I would like to dive deeper into the world lock logic so my team and I are able to figure out the best method to fix the misalignment bugs.

Please let me know if you are open for us to set up a short meeting where we can show what is going on in the editor while we have multiple headsets running at the same time.

Thank you,
David Montecalvo (VirtualGo LLC)

[Squashi11]

Misalignment will always be an issue on these platforms. "Scene Drifts" often occurs when as user has moved too much. Or is standing static in the scene. You can post some pictures which may help us solve the issue of what you are experiencing

[TudorJude]

Hello,

What API are you using? It sounds to me like you are creating some kind of colocation experience and then using world locking to align everyone. Let's try and debug the issue here as much as we can for now.

[djmonte94]

Hi TudorJude,

Thanks for jumping in!

To clarify, we aren't building a standard co-located experience using Shared Spatial Anchors. This is a remote asymmetric experience using MRUK.

The Host is in Mixed Reality, physically moving through their scanned environment with World Lock enabled. The remote VR players download a serialized JSON of that MRUK scene and navigate it remotely.

The Core Issue:
Because World Lock dynamically shifts the Host's OVRCameraRig to combat tracking drift, their global Unity coordinates change even when they are standing still. This makes standard networked position syncing fail, causing the Host to appear misaligned or sliding relative to the remote players' downloaded room.

I completely understand wanting to debug here on the forum first, but this specific tracking space mismatch is much easier to diagnose when seeing the MR and VR clients side-by-side. Would you be open to a brief video call with our team later this week or next so I can show you the exact behavior in the Unity Editor? Thank you

[TudorJude]

Hello,

Unfortunately, we can't facilitate a call at this time. I do think we can solve the problem here, however.

If I understand correctly, the host is in MR, using MRUK, World Locking on. The clients are in VR, download the host's room on their device and then they can navigate in the room that you sent them. Is this correct so far? If you could add some videos of the experience here, it would also be helpful. I do see the problem here, if you are sending the OVRCameraRig's transform through the network, it will constantly move because world locking will drift it. We drift the tracking space object specifically, in Unity.

If the experience was colocated, you could have all headsets sync to a specific spatial anchor using "SetCustomWorldLockAnchor" and sharing a spatial anchor to everyone. From your message, it seems that this is a remote experience so we can figure out a different solution there.

Here's some context on how it works:

Under the hood, we spawn a spatial anchor and then persist it (this is done to have it survive loss of tracking). You can read more about anchors here:
https://developers.meta.com/horizon/documentation/unity/unity-spatial-anchors-overview/
After that, we “world lock” against the “currentWorldLockAnchor” which is chosen based on the following priority:

• A customWorldLockAnchor that can be set in engine (usually used for colocation)

• If a room is loaded, the closest floor (for scene v1) or room (for scenev2) anchor

• If none of the above are available, we use the spatial anchor that we spawned.

We calculate a world lock offset between the base space and the currentWorldLockAnchor. The pose of the currentWorldLockOffset anchor is considered a “source of truth” since we call xrLocateSpace every frame for this anchor. These anchors should remain “locked” to the physical word according to their OpenXR spec. The base space, however, can drift or move. The delta between these poses is then used to offset the tracking space when World Locking is enabled.

This world lock offset is calculated in our shared library, which is a C++ library so you can't see the code inside it but for world locking, this is pretty much it. Get a spatial anchor and because spatial anchors are designed to stay "in place in the physical world" you can always get the offset between that and the base space (0,0,0) every frame.

Here's a simple example. Spawn a spatial anchor and record its initial tracking pose after it localizes.

Then put this code in an update function

            var currentTrackingPose = _spatialAnchor.transform.ToTrackingSpacePose(_cachedCamera); // cached camera is just the main cam
            var delta = _initialTrackingPose * currentTrackingPose.Inverse();
// we really only want to apply the offset in such a way that we don't tilt the room so we only take yaw into account
            var yawOnly = Quaternion.Euler(0f, delta.orientation.eulerAngles.y, 0f);
            _deltaPose = new Pose(delta.position, yawOnly);

That delta pose is what you would use to offset your tracking space. You can, for example, disable MRUK world locking and use this. Send the camera pose without this delta and then apply the delta yourself. The reason we move the camera and not the objects themselves is because it would, for example, break physics and there would be other problems, but you could potentially move the virtual content with this offset. It will still appear to be in the correct position but now the virtual content will move around in your world space so I don't know if that will be much better for you.

Hope this helps