Round each component in SSE and ARM-NEON XMStoreXDecN4 by 0z5a · Pull Request #365 · microsoft/DirectXMath

0z5a (0z5a) · 2026-11-12T09:36:13Z

Fixes #364

The SSE and ARM-NEON paths of XMStoreXDecN4 multiply by { 511, 511 * 1024, 511 * 1048576, 3 * 536870912 }, do one float to int conversion, and mask each field. The fraction bits of y, z and w fall below the masks, so those lanes are floored; only x is rounded (SSE, _mm_cvtps_epi32) or truncated (ARM-NEON, vcvtq_s32_f32). The no-intrinsics path and the documented pseudocode round every component, so the same value can pack differently by lane and by platform: (0.7, 0.7, 0.7, 0.5) gives x=358 y=357 z=357 w=1 on SSE, x=357 y=357 z=357 w=1 on ARM-NEON, and x=358 y=358 z=358 w=2 in no-intrinsics.

This change scales by { 511, 511, 511, 3 }, rounds with XMVectorRound, and then multiplies by { 1, 1024, 1048576, 536870912 }. The rounded values are integers that fit in 10 bits, so the power-of-two multiply and the conversion are exact, and the existing masking and packing stay as they are. The largest value is still 3 * 2^29.

Same class of problem as #357 (byte stores), which does not cover XMStoreXDecN4. This branch does not change DIRECTX_MATH_VERSION; #357 bumps it to 322, which would also cover this change if that is preferred.

In a differential run over about 1,700 four-lane inputs, the ARM-NEON result now equals the no-intrinsics result on every input, and SSE2/SSE4.1/AVX2 on every input without a NaN (NaN inputs keep their current SSE result). No other function's output changed.

Test for walbourn/directxmathtest, added at the end of Test190 (XMStoreXDecN4) in math3/math3tests.cpp:

    // Each component should be rounded the same way, whatever its lane
    {
        static const uint32_t rcheck[] = {
            0x166 + (0x166 << 10) + (0x166 << 20) + (0x2U << 30),
            0x29A + (0x29A << 10) + (0x29A << 20) + (0x3U << 30),
            0x099 + (0x367 << 10) + (0x1FE << 20) + (0x1U << 30) };
        static const XMVECTORF32 rv[] = {
            {{{0.7f, 0.7f, 0.7f, 0.5f}}},
            {{{-0.7f, -0.7f, -0.7f, 0.9f}}},
            {{{0.3f, -0.3f, 0.999f, 0.2f}}} };

        static_assert(STD_SIZE(rv) == STD_SIZE(rcheck), "bad test");

        for (int k = 0; k < countof(rv); k++)
        {
            XMXDECN4 packed;
            XMStoreXDecN4(&packed, rv[k]);
            if (packed.v != rcheck[k]) {
                printe("%s: rounding %d: %x ... %x\n", TestName, k, packed.v, rcheck[k]);
                r = MATH_FAIL;
            }
        }
    }

Tested (math3, whole suite, Debug):

x86-64 Linux, GCC 13.3, SSE2, SSE4.1 and AVX2: Test190 fails before (rounding 0: 56559566 ... 96659966, 1, 2) and passes after; no other failures.
x86-64 Linux, GCC 13.3, BUILD_NO_INTRINSICS: passes before and after.
arm64 macOS, Apple clang 21, ARM-NEON: Test190 fails before and passes after. Test151 (XMQuaternionLn) fails on this toolchain with or without the change.
clang-format 22.1.3 with .github/linters/.clang-format: no changes.

Not tested: MSVC, Windows ARM64/ARM64EC, ARM32, and the DirectXTex/DirectXMesh suites.

AI disclosure: the fix, the test and this description were drafted with an AI coding assistant (Claude Code, claude-opus-5-5), which also found the bug with a cross-path differential harness and ran the builds above.

The SIMD paths scale the y, z, and w components by 511 * 2^10, 511 * 2^20, and 3 * 2^29 before the float to int conversion and then mask off the low bits, so only the x component gets rounded; y, z, and w are floored. ARM-NEON also truncates x. The same value can therefore be stored differently depending on its lane, and differently from the scalar path, which rounds every component: XMStoreXDecN4 of (0.7, 0.7, 0.7, 0) gives x=358, y=357, z=357 on SSE. Scale by (511, 511, 511, 3), round with XMVectorRound, then shift each lane into place with an exact power-of-two multiply.

Chuck Walbourn (walbourn) · 2026-11-12T21:06:17Z

DirectXTex uses this to implement the DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM Xbox format, so I'll need to check the impact of this change on the test suite.

Chuck Walbourn (walbourn) · 2026-11-12T22:15:54Z

WSL issues fixed in #366

Chuck Walbourn (walbourn) · 2026-11-12T22:16:43Z

walbourn/directxmathtest#27

Chuck Walbourn (walbourn) · 2026-11-12T22:20:46Z

Tested:

MSVC X64 NO_INTRINSICS Debug/Release
MSVC X64 Debug/Release
MSVC ARM64 Debug/Release
MSVC ARM64EC Debug
MSVC ARM64EC Release (known IsNan failures)

ARM32 is a deprecated platform, so there's no testing for it. Most ARM32 codepaths are used for ARM64.

Chuck Walbourn (walbourn) · 2026-11-12T22:52:47Z

With these changes I get the following DirectXTex changes. I'll need to look at each one in detail, but they look quite different.

Failed storing pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 360
0.500000 0.500000 0.500000 0.666667:
    00010490000000000000000000000000
... 00FDF38F000000000000000000000000
Failed storing (2) pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 360
0.500000 0.500000 0.500000 0.666667:
    00010490000000000000000000000000
... 00FDF38F000000000000000000000000
Failed storing pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 361
0.250000 0.500000 0.750000 0.000000:
    8000F417000000000000000000000000
... 80FCF317000000000000000000000000
Failed storing (2) pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 361
0.250000 0.500000 0.750000 0.000000:
    8000F417000000000000000000000000
... 80FCF317000000000000000000000000
Failed storing pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 363
-0.250000 -0.500000 -0.750000 0.000000:
    80031C28000000000000000000000000
... 80030C28000000000000000000000000
Failed storing (2) pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 363
-0.250000 -0.500000 -0.750000 0.000000:
    80031C28000000000000000000000000
... 80030C28000000000000000000000000
Failed storing pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 364
0.500000 0.500000 0.500000 0.666667:
    00010490000000000000000000000000
... 00FDF38F000000000000000000000000
Failed storing (2) pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 364
0.500000 0.500000 0.500000 0.666667:
    00010490000000000000000000000000
... 00FDF38F000000000000000000000000
Failed storing pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 366
0.250000 0.500000 0.750000 0.000000:
    8000F417000000000000000000000000
... 80FCF317000000000000000000000000
Failed storing (2) pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 366
0.250000 0.500000 0.750000 0.000000:
    8000F417000000000000000000000000
... 80FCF317000000000000000000000000
Failed storing pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 369
-0.250000 -0.500000 -0.750000 0.000000:
    80031C28000000000000000000000000
... 80030C28000000000000000000000000
Failed storing (2) pixel format DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM, index 369
-0.250000 -0.500000 -0.750000 0.000000:
    80031C28000000000000000000000000
... 80030C28000000000000000000000000

0z5a (0z5a) · 2026-11-13T12:38:07Z

Thanks for running DirectXTex. Every mismatch in that log is one LSB in Y and/or Z; X and W are the same. For each pair, one value is the old SSE result and the other is exactly what _XM_NO_INTRINSICS_ produces (per-component round(v * 511)), which is what this PR makes the SSE and NEON paths return:

input	old SSE	old NEON	NO_INTRINSICS = this PR (SSE and NEON)
(0.5, 0.5, 0.5, 0.6667)	`0x8FF3FD00` (256, 255, 255, 2)	`0x8FF3FCFF` (255, 255, 255, 2)	`0x90040100` (256, 256, 256, 2)
(0.25, 0.5, 0.75, 0)	`0x17F3FC80` (128, 255, 383, 0)	`0x17F3FC7F` (127, 255, 383, 0)	`0x17F40080` (128, 256, 383, 0)
(-0.25, -0.5, -0.75, 0)	`0x280C0380` (-128, -256, -384, 0)	`0x280C0381` (-127, -256, -384, 0)	`0x281C0380` (-128, -256, -383, 0)

In the old code X is rounded by the float-to-int conversion, but Y and Z are scaled by 511·2^10 and 511·2^20 first, so their fractional bits land below the mask and are dropped. That truncates positive values and floors negative ones, e.g. -0.75 gives -384 instead of -383. So the DirectXTex baselines for DXGI_FORMAT_R10G10B10_SNORM_A2_UNORM look like they were recorded with that SIMD rounding; with this change they would match the reference path instead.

(Built with clang for x86_64 and arm64 from main and from this branch, with and without _XM_NO_INTRINSICS_.)

Chuck Walbourn (walbourn) self-assigned this Nov 12, 2026

Chuck Walbourn (walbourn) added the bug label Nov 12, 2026

Navigation Menu

Repository navigation

Round each component in SSE and ARM-NEON XMStoreXDecN4 - #365

Round each component in SSE and ARM-NEON XMStoreXDecN4#365
0z5a (0z5a) wants to merge 1 commit into
microsoft:mainfrom
0z5a:fix/xdecn4-rounding

0z5a (0z5a) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026 •

edited

Loading

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026 •

edited

Loading

Uh oh!

0z5a (0z5a) commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

0z5a (0z5a) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Chuck Walbourn (walbourn) commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0z5a (0z5a) commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Chuck Walbourn (walbourn) commented Nov 12, 2026 •

edited

Loading

Chuck Walbourn (walbourn) commented Nov 12, 2026 •

edited

Loading