Skip to content

environments comparison splits one environment into several sides when boot_time drifts (WSL2) #121

Closed

Description

@MarcoGorelli

Spotted during benchx-dev/benchx_examples#11

What happens

On WSL2, bx compare --profile environments refuses comparisons that are fine:

run pandas-1795022142  profile environments  baseline pandas-2.3.3  contender None

failed invariants (1)
  {'invariant': 'two-sides', 'detail': '3 sides: [\'"pandas-2.3.3"\', \'"pandas-2.3.3"\', \'"pandas-3.1.1"\']'}

Reproducer: examples/pandas-asv/run.sh in benchx-examples, run on WSL2. Each venv is used for every call on its side, yet benchx finds 3 or 4 sides instead of 2, and the count changes from run to run.

Cause

Results that share a label value get different coordinates.environment.system.os.boot_time values. In one run:

  • .venv-a results: 2026-11-16 16:28:41 and 2026-11-16 16:28:39
  • .venv-b results: 2026-11-16 16:28:39, and one with no boot_time at all

pyperf reads btime from /proc/stat. On WSL2 that value follows time.time() - uptime, so it moves: reading it once a second, I saw it go from 1794846221 to 1794846220 within a few seconds. pyperf keeps a key in the file-level metadata only if every run has the same value (_common_metadata). So when btime moves between the processes of one call, boot_time is left out, which is where the missing value comes from.

src/benchx/adapters/pyperf.py (_SYSTEM) maps boot_time to environment.system.os.boot_time. The environments profile in src/benchx/compare.py uses all of /coordinates/environment/system as the side's identity. So any drift in boot_time creates a new side. The comment above _SYSTEM says "the default identity policy ignores cpu and os", but the environments projection doesn't ignore them.

Both are unchanged on main as of 04f550b. Removing boot_time from the result files and re-ingesting gives the expected result, with both a checkout (36280f7) and benchx 0.1.0 from PyPI:

run pandas-1795019409  profile environments  baseline pandas-2.3.3  contender pandas-3.1.1
  differs: coordinates.environment.system.deps.pandas, coordinates.environment.system.deps.sha256
IMPROVED (1)
  strings.Methods.time_lower       wall-time  effect  -89.05%  noise   0.94%  rounds=3

Why it matters beyond WSL2

Boot time describes when the machine started, not what software is being compared. I've only seen this on WSL2. From the code, though, anything that changes btime between rounds would split sides the same way on any platform, such as a reboot or possibly a clock step.

Possible fixes

  • Leave boot_time out of environment.system and keep it in provenance info. It is useful for diagnosis, but it shouldn't identify an environment. Or
  • Leave os.boot_time out of the environments profile's projection, or out of identity in general.

Activity

  1. added a commit that references this issue on Nov 19, 2026
    1dbec41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions