Skip to content

VR: tracking with weapons drawn (other players' VRIK body pose) #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cec339 wants to merge 5 commits into
base: vr-support
Choose a base branch
from
Draft

VR: tracking with weapons drawn (other players' VRIK body pose) #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
60d1e99
feat: server setting bUseLegacyHandPose for VR pose sync
cec339 Sep 28, 2026
edb057e
feat: vr full-body tracking for other players
cec339 Sep 28, 2026
91c87fb
fix: remove synthesised vr melee attack action
cec339 Oct 3, 2026
aba2981
fix: vr menus do not send the menu pose as the player's body
cec339 Oct 3, 2026
80a04c6
fix: body-mode fallback leaves a drawn weapon to the animation
cec339 Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 0 additions & 118 deletions Code/client/Games/Skyrim/Actor.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -983,130 +983,12 @@ bool TP_MAKE_THISCALL(HookSpawnActorInWorld, Actor)
TP_THIS_FUNCTION(TDamageActor, bool, Actor, float aDamage, Actor* apHitter, bool aKillMove);
static TDamageActor* RealDamageActor = nullptr;

#if TP_SKYRIMVR
// A VR melee swing never runs an action, so other players never see one.
//
// Remote clients only ever see an attack because HookPerformAction turns the local actor's
// ActorMediator::PerformAction into an ActionEvent that they replay through ForceAction. On the
// desktop game a swing is an action: input runs ActionRightAttack, which runs PerformAction, which
// drives the animation graph. VR drives the swing from controller motion instead, and a run of the
// probes that used to live here settled it: over five real melee hits, PerformAction was not called
// once. The seven ActionLeftAttack calls in the same session all landed more than two minutes away
// from any hit, so they are some other input, not the swing.
//
// So the action is synthesised here, at the one moment the game agrees the swing connected. It is
// built exactly the way HookPerformAction builds one and goes out through the same runner event, so
// it travels over the existing wire format and replays through ForceAction like any other attack.
// Nothing in the protocol, the encoding or the server changes.
//
// Two things this cannot do, both inherent to using the hit rather than the swing:
// - the animation starts when the blow lands, so remotes see it without its wind-up
// - a miss produces nothing, because a miss never reaches this hook
// Both would be fixed by hooking VR's swing detection instead, which was never located.
//
// The same treatment for bow and crossbow shots was tried and removed. Projectile::Launch is a
// better signal than this one, being the release rather than the hit, and the arrow already syncs
// from there, but no bow event the graph offers would play on a remote actor: `arrowRelease` and
// `bowDrawStart` were both refused, and `attackStart` played a melee swing. That was with
// iRightHandType reading 7 on the receiver, so the remote graph knew it held a bow and refused
// anyway. The unexplored lead is that the remote copy may not really have the bow equipped, which
// would make it an equipment sync problem rather than an animation one.
//
// The filter exists because damage over time arrives here too, attributed to the same hitter. In
// the measured run a real hit was a single call of 13 to 18 damage, while a bleed was 288 calls
// over four seconds decaying from 18 to 0.03, one per frame. A weapon hit and a per-frame tick are
// three orders of magnitude apart, and no swing repeats within a third of a second, so a floor plus
// a rate limit separates them. Both are heuristics; the exact discriminator is the game's own
// TESHitEvent, which carries the weapon and the power/bash flags, and which nothing in the client
// subscribes to yet.
static constexpr float kVRMeleeMinDamage = 1.0f;
static constexpr uint64_t kVRMeleeMinInterval = 300; // ms

static void SynthesiseVRMeleeAction(Actor* apHitter, Actor* apHittee, float aDamage) noexcept
{
if (aDamage < kVRMeleeMinDamage)
return;

const uint64_t cTick = World::Get().GetTick();
static uint64_t s_lastTick = 0;
if (cTick - s_lastTick < kVRMeleeMinInterval)
return;

// ActionLeftAttack, which is the only attack action SkyrimVR itself ever performs: across a
// session of probing the game ran it seven times and ActionRightAttack not once, whatever hand
// the weapon was in.
constexpr uint32_t kActionLeftAttack = 0x13004;
const uint32_t cActionId = kActionLeftAttack;

ActionEvent action;
action.Tick = cTick;
action.ActorId = apHitter->formID;
action.ActionId = cActionId;
// Not the actor that was hit. A form id like 0xFF0008D1 is a temporary reference, allocated per
// session per machine, so it names nothing on the receiving client: the probes caught the
// sender writing FF00087A and the receiver resolving FF0008D1 to null. Every action the game
// itself performs arrives here with a target of 0, so the synthesised one matches that.
action.TargetId = 0;
action.State1 = apHitter->actorState.flags1;
action.State2 = apHitter->actorState.flags2;
action.Type = 2; // what the game itself passes as unkInput for an attack, with someFlag clear
// This is the field that makes the whole thing work, so do not drop it. The receiver copies it
// into TESActionData::eventName, and ForceAction has nothing to play without it: while this was
// empty the synthesised attack was refused 29 times out of 29, and filling it is what turned
// that into a visible swing. Both names are the behaviour graph's own, read out of VR's
// meshes\actors\character\behaviors\0_master.hkx. Slot 1 is the right hand, per
// Actor::GetEquippedWeapon.
action.EventName = apHitter->GetEquippedWeapon(1) ? "attackStart" : "attackStartLeftHand";
apHitter->SaveAnimationVariables(action.Variables);

// Every source of damage reaches this hook, so a crossbow bolt landed a melee swing on the other
// client. Nothing in the arguments says where the damage came from, and the return address does
// not either: melee and projectile hits were logged arriving from the same call site,
// 0x14062F60C, so DamageActor has one caller and cannot be told apart that way. Range was tried
// and is no good, since the reported bow hits landed at 262 to 363 units, inside any sane melee
// gate.
//
// What does separate them is the weapon in hand, and the graph already carries it.
// iRightHandType and iLeftHandType hold Skyrim's weapon animation type, and they are entries 5
// and 4 of the descriptor's integer table that SaveAnimationVariables has just filled:
// 0 hand to hand, 1 sword, 2 dagger, 3 axe, 4 mace, 5 two-handed sword,
// 6 two-handed axe or hammer, 7 bow, 8 staff, 9 crossbow
// So a swing is anything at or below 6. The test is written that way round on purpose: if the
// enum is off, melee coverage narrows, which is a great deal better than firing a swing on every
// shot. Confirmed in play against an IronWarhammer and a LongBow.
constexpr size_t kLeftHandTypeIndex = 4;
constexpr size_t kRightHandTypeIndex = 5;
constexpr uint32_t kMaxMeleeAnimationType = 6;

const auto& cIntegers = action.Variables.Integers;
if (cIntegers.size() <= kRightHandTypeIndex)
return; // SaveAnimationVariables found no descriptor, so nothing can be told about the hands

const uint32_t cRightHandType = cIntegers[kRightHandTypeIndex];
const uint32_t cLeftHandType = cIntegers[kLeftHandTypeIndex];

if (cRightHandType > kMaxMeleeAnimationType && cLeftHandType > kMaxMeleeAnimationType)
return;

// Only once a swing is committed to, so a suppressed shot does not spend the slot and mute the
// real hit that follows it.
s_lastTick = cTick;

World::Get().GetRunner().Trigger(action);
}
#endif

// TODO: this is flawed, since it does not account for invulnerable actors
bool TP_MAKE_THISCALL(HookDamageActor, Actor, float aDamage, Actor* apHitter, bool aKillMove)
{
if (apHitter)
World::Get().GetRunner().Trigger(HitEvent(apHitter->formID, apThis->formID));

#if TP_SKYRIMVR
if (apHitter && apHitter->formID == 0x14)
SynthesiseVRMeleeAction(apHitter, apThis, aDamage);
#endif

float realDamage = GameplayFormulas::CalculateRealDamage(apThis, aDamage, aKillMove);

float currentHealth = apThis->GetActorValue(ActorValueInfo::kHealth);
Expand Down
254 changes: 254 additions & 0 deletions Code/client/Games/Skyrim/Havok/BodyReferenceSkeleton.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
#include <TiltedOnlinePCH.h>
#include <Havok/BodyReferenceSkeleton.h>
#include <Actor.h>
#include <BSAnimationGraphManager.h>
#include <NetImmerse/NiNode.h>
#include <cstring>
#include <limits>

namespace BodyTracking
{
#if TP_SKYRIMVR
namespace
{
// VR 1.4.15 reflection records (pinned image SHA 8daf0519...46d4):
// hkbCharacter.setup +50; hkbCharacterSetup.animationSkeleton +20;
// hkaSkeleton parentIndices +18, bones +28, referencePose +38.
// BShkbAnimationGraph character +C0, holder +210, root +218 are separately
// verified against the VR constructor and the pinned CommonLib layout.
// See round4-member-records.json and the O3 build document for full provenance.
struct HkArrayView { uintptr_t Data; int32_t Size; uint32_t CapacityAndFlags; };
struct GraphView
{
uint8_t Prefix[0x110];
uintptr_t Setup;
uint8_t ToHolder[0x210 - 0x118];
uintptr_t Holder;
uintptr_t Root;
};
struct SetupView { uint8_t Prefix[0x20]; uintptr_t AnimationSkeleton; };
struct SkeletonView
{
uint8_t Prefix[0x18];
HkArrayView Parents, Bones, ReferencePose;
};
struct BoneView { uintptr_t Name; uint8_t Locked; uint8_t Padding[7]; };
struct PoseView { float Translation[4], Rotation[4], Scale[4]; };
static_assert(sizeof(HkArrayView) == 0x10 && sizeof(BoneView) == 0x10 && sizeof(PoseView) == 0x30);
static_assert(offsetof(GraphView, Setup) == 0x110 && offsetof(GraphView, Holder) == 0x210 && offsetof(GraphView, Root) == 0x218);
static_assert(offsetof(SetupView, AnimationSkeleton) == 0x20);
static_assert(offsetof(SkeletonView, Parents) == 0x18 && offsetof(SkeletonView, Bones) == 0x28 && offsetof(SkeletonView, ReferencePose) == 0x38);

// Cache queried memory regions only for this locked snapshot, not between
// frames. Names are copied in region-sized chunks, avoiding a syscall/byte.
class BoundedRead
{
public:
bool Copy(uintptr_t aSource, void* apDestination, size_t aBytes)
{
if (!aSource || aBytes > std::numeric_limits<uintptr_t>::max() - aSource) return false;
auto* destination = static_cast<uint8_t*>(apDestination);
while (aBytes)
{
uintptr_t end{};
if (!RegionEnd(aSource, end)) return false;
const auto count = std::min(aBytes, static_cast<size_t>(end - aSource));
std::memcpy(destination, reinterpret_cast<const void*>(aSource), count);
destination += count;
aSource += count;
aBytes -= count;
}
return true;
}
template<class T> bool Read(uintptr_t aSource, T& aValue) { return Copy(aSource, &aValue, sizeof(T)); }
bool Name(uintptr_t aSource, std::string& aName, bool aTagged = true)
{
// hkStringPtr's low bit records owned storage, not part of the address.
if (aTagged) aSource &= ~uintptr_t{1};
aName.clear();
std::array<char, kMaxBoneNameBytes> text{};
size_t copied{};
while (copied < text.size())
{
uintptr_t end{};
if (!RegionEnd(aSource, end)) return false;
const auto count = std::min(text.size() - copied, static_cast<size_t>(end - aSource));
std::memcpy(text.data() + copied, reinterpret_cast<const void*>(aSource), count);
const auto* terminator = static_cast<const char*>(std::memchr(text.data() + copied, 0, count));
if (terminator)
{
aName.assign(text.data(), static_cast<size_t>(terminator - text.data()));
return !aName.empty();
}
copied += count;
aSource += count;
}
return false;
}
private:
bool RegionEnd(uintptr_t aSource, uintptr_t& aEnd)
{
if (!aSource) return false;
for (const auto& region : Regions)
if (aSource >= region.first && aSource < region.second) { aEnd = region.second; return true; }
MEMORY_BASIC_INFORMATION info{};
if (!VirtualQuery(reinterpret_cast<const void*>(aSource), &info, sizeof(info)) ||
info.State != MEM_COMMIT || (info.Protect & (PAGE_NOACCESS | PAGE_GUARD))) return false;
const auto access = info.Protect & 0xff;
if (access != PAGE_READONLY && access != PAGE_READWRITE && access != PAGE_WRITECOPY &&
access != PAGE_EXECUTE_READ && access != PAGE_EXECUTE_READWRITE && access != PAGE_EXECUTE_WRITECOPY) return false;
const auto start = reinterpret_cast<uintptr_t>(info.BaseAddress);
if (info.RegionSize > std::numeric_limits<uintptr_t>::max() - start) return false;
aEnd = start + info.RegionSize;
if (aEnd <= aSource) return false;
Regions[Next++ % Regions.size()] = {start, aEnd};
return true;
}
std::array<std::pair<uintptr_t, uintptr_t>, 16> Regions{};
size_t Next{};
};

bool ValidArray(const HkArrayView& aArray, int32_t aCount)
{
return aArray.Data && aCount > 0 && aCount <= static_cast<int32_t>(kMaxReferenceBones) &&
aArray.Size == aCount && (aArray.CapacityAndFlags & 0x3fffffffU) >= static_cast<uint32_t>(aCount);
}

struct ManagerLease
{
BSAnimationGraphManager* Pointer{};
~ManagerLease() { if (Pointer) Pointer->Release(); }
};
}
#endif

ReferenceFailure ReadBodyReferenceSkeleton(Actor& aActor, NiNode* apExpectedRoot,
uint64_t aGeneration, ReferenceSkeleton& aOutput,
ReferenceDiagnostic* apDiagnostic, bool aReuse)
{
bool accepted{};
struct ClearOnFailure
{
ReferenceSkeleton& Output;
bool& Accepted;
~ClearOnFailure() { if (!Accepted) Output = {}; }
} clear{aOutput, accepted};
if (apDiagnostic) *apDiagnostic = {0, 0, reinterpret_cast<uintptr_t>(&aActor),
reinterpret_cast<uintptr_t>(&aActor.animationGraphHolder), reinterpret_cast<uintptr_t>(apExpectedRoot)};
#if !TP_SKYRIMVR
return ReferenceFailure::UnsupportedRuntime;
#else
if (!apExpectedRoot || !aGeneration || aActor.GetNiNode() != apExpectedRoot) return ReferenceFailure::ActorIdentity;
// Acquire in the actor update phase; retain the root while the additional
// graph/setup/skeleton view is followed. This does not claim that a raw
// pointer can be safely acquired from an arbitrary game thread.
struct RootLease
{
NiNode* Root;
~RootLease() { Root->DecRef(); }
} rootLease{apExpectedRoot};
apExpectedRoot->IncRef();
ManagerLease lease{};
const bool acquired = aActor.animationGraphHolder.GetBSAnimationGraph(&lease.Pointer);
// Release a non-null result even when the holder reports failure.
if (!acquired || !lease.Pointer) return ReferenceFailure::Manager;
ReferenceSkeleton snapshot{};
{
// The lock dies before the retained manager on EVERY return path.
BSScopedLock<BSRecursiveLock> lock{lease.Pointer->lock};
auto& manager = *lease.Pointer;
const auto& graphs = manager.animationGraphs;
const auto index = manager.animationGraphIndex;
if (!graphs.size || graphs.size > 64 || index >= graphs.size) return ReferenceFailure::Graph;
BoundedRead read;
std::vector<GraphView> views(graphs.size);
std::vector<ReferenceGraphIdentity> identities(graphs.size);
if (apDiagnostic)
{
apDiagnostic->GraphCount = graphs.size;
apDiagnostic->ActiveGraph = index;
apDiagnostic->Graphs.resize(graphs.size);
}
if (graphs.capacity < 0 && (graphs.size != 1 || index != 0)) return ReferenceFailure::Graph;
if (graphs.capacity >= 0 && graphs.capacity < static_cast<int32_t>(graphs.size)) return ReferenceFailure::Graph;
const auto array = reinterpret_cast<uintptr_t>(graphs.data);
if (graphs.capacity >= 0 && array > UINTPTR_MAX - graphs.size * sizeof(uintptr_t)) return ReferenceFailure::Graph;
for (uint32_t slot = 0; slot < graphs.size; ++slot)
{
uintptr_t address{};
const bool slotRead = graphs.capacity < 0 ? (address = array, true) : read.Read(array + slot * sizeof(uintptr_t), address);
auto& identity = identities[slot];
identity.Address = address;
identity.Readable = slotRead && read.Read(address, views[slot]);
if (identity.Readable)
{
identity.Holder = views[slot].Holder;
identity.Root = views[slot].Root;
}
if (apDiagnostic) apDiagnostic->Graphs[slot].Identity = identity;
}
const int selected = SelectReferenceGraph(identities, reinterpret_cast<uintptr_t>(&aActor), reinterpret_cast<uintptr_t>(apExpectedRoot));
if (apDiagnostic)
{
apDiagnostic->SelectedGraph = selected;
const auto& reported = identities[selected >= 0 ? selected : index];
apDiagnostic->StoredHolder = reported.Holder;
apDiagnostic->StoredRoot = reported.Root;
// Names are diagnostic only; a missing name cannot select a graph.
if (selected < 0) for (auto& item : apDiagnostic->Graphs)
{
uintptr_t name{};
if (!item.Identity.Readable || item.Identity.Root > UINTPTR_MAX - 0x10 ||
!read.Read(item.Identity.Root + 0x10, name) || !read.Name(name, item.RootName, false)) item.RootName = "<unreadable>";
}
}
if (selected < 0) return selected == -2 ? ReferenceFailure::Graph : ReferenceFailure::ActorIdentity;
const auto graphAddress = identities[selected].Address;
const auto& graph = views[selected];
SetupView setup{};
SkeletonView skeleton{};
if (!read.Read(graph.Setup, setup) || !read.Read(setup.AnimationSkeleton, skeleton)) return ReferenceFailure::Unreadable;
const auto count = skeleton.Bones.Size;
if (!ValidArray(skeleton.Parents, count) || !ValidArray(skeleton.Bones, count) || !ValidArray(skeleton.ReferencePose, count))
return ReferenceFailure::Arrays;
if (aReuse && aOutput.Generation == aGeneration && aOutput.GraphIdentity == graphAddress &&
aOutput.SkeletonIdentity == setup.AnimationSkeleton && aOutput.RootIdentity == graph.Root &&
aOutput.ParentsIdentity == skeleton.Parents.Data && aOutput.BonesIdentity == skeleton.Bones.Data &&
aOutput.PoseIdentity == skeleton.ReferencePose.Data && aOutput.Bones.size() == static_cast<size_t>(count) &&
aActor.GetNiNode() == apExpectedRoot)
{
accepted = true;
return ReferenceFailure::None;
}
std::vector<int16_t> parents(count);
std::vector<BoneView> bones(count);
std::vector<PoseView> poses(count);
if (!read.Copy(skeleton.Parents.Data, parents.data(), parents.size() * sizeof(int16_t)) ||
!read.Copy(skeleton.Bones.Data, bones.data(), bones.size() * sizeof(BoneView)) ||
!read.Copy(skeleton.ReferencePose.Data, poses.data(), poses.size() * sizeof(PoseView))) return ReferenceFailure::Unreadable;
snapshot.Bones.resize(count);
for (int32_t i = 0; i < count; ++i)
{
auto& bone = snapshot.Bones[i];
if (!read.Name(bones[i].Name, bone.Name) || bones[i].Locked > 1) return ReferenceFailure::Names;
bone.Parent = parents[i];
bone.TranslationLocked = bones[i].Locked != 0;
if (!ReferenceTransform(poses[i].Translation, poses[i].Rotation, poses[i].Scale, bone.Local))
return ReferenceFailure::Transform;
}
if (const auto failure = ValidateReferenceSkeleton(snapshot); failure != ReferenceFailure::None) return failure;
if (aActor.GetNiNode() != apExpectedRoot) return ReferenceFailure::ActorIdentity;
snapshot.GraphIdentity = graphAddress;
snapshot.SkeletonIdentity = setup.AnimationSkeleton;
snapshot.RootIdentity = graph.Root;
snapshot.ParentsIdentity = skeleton.Parents.Data;
snapshot.BonesIdentity = skeleton.Bones.Data;
snapshot.PoseIdentity = skeleton.ReferencePose.Data;
snapshot.Generation = aGeneration;
}
aOutput = std::move(snapshot);
accepted = true;
return ReferenceFailure::None;
#endif
}
}
Loading
Loading