Skip to content

Make CI's skipped tests visible, and stop reporting zero-test files as passes - #12

Closed
johng wants to merge 3 commits into
robertsdotpm:mainfrom
johng:worktree-ci-skipped-tests
Closed

johng wants to merge 3 commits into
robertsdotpm:mainfrom
johng:worktree-ci-skipped-tests

Conversation

@johng

@johng johng commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Started from "why are tests skipped on CI -- flaky perf work on shared boxes,
or real issues?" It is neither: the skips are mostly legitimate platform
gates, but CI had no way to tell those apart from tests that were not running
for fixable reasons, and several were not running at all.

What was wrong

Files that execute zero tests reported PASS. A file where every
collected test skips exits 0, so classify() called it a pass and the tally
counted it as one. _summary_line() did not match "skipped", so
"10 skipped in 0.70s" matched nothing and the file printed with a blank
summary -- indistinguishable from one that ran. On run 35022043268, green
throughout:

leg files that executed no tests
ubuntu / 3.14 14 (reported as "236 passed, 0 skipped")
ubuntu / 3.13 16
macOS / 3.14 26

tests/test_greenlet_interop.py has never run on CI. It is in that set on
every leg. CLAUDE.md names it the guard for the greenlet/TLBC coexistence
invariant. Nothing installs greenlet, so all 10 of its tests skip -- and it
looks green. Verified: install greenlet (3.5.6, cp314t wheel) and it is
10 passed on 3.14t.

tests/test_chan_properties.py is skipped on both 3.13 legs for a fixable
reason.
The comment blamed PyO3's lack of free-threaded support below 3.14 --
true, but only of the newest hypothesis. 6.156.1 is the first release with a
Rust core (the first with platform wheels instead of py3-none-any); 6.155.7
and earlier are pure Python. Verified: with hypothesis<6.156 the file is
4 passed on 3.13t (4 property tests x 200 examples over channel FIFO /
conservation / close / select). Same shape for cryptography + trustme, which
gate 5 TLS tests and do publish cp314t wheels.

The contention valve was inverted. 38 pytest.skip("... timed out (shared-box contention)") sites across 22 files turn a hung child into a
silent skip. Their deadlines were literals -- the only deadlines in the suite
ignoring RUNLOOM_TIMEOUT_MULT, which CI sets to 2 precisely because its
runners are slow. So the scaler doubled the outer per-file ceiling and left
the inner ones fixed: on the box the scaler exists for, the inner one trips
first and the test skips instead of running slower. I found no evidence of it
firing in the runs I sampled -- this closes the mechanism, not an observed
failure.

What this changes

  • tests/run_isolated.py -- ALLSKIP, the missing third verdict; -rs so
    skip reasons reach the output; two unconditional end-of-run blocks (the
    files that executed nothing, and a census of skipped tests by reason); the
    tally now reads N passed, M ran nothing, K not-passed.
  • tools/ci/test_patched_cpython.sh -- best-effort install of
    greenlet / hypothesis / cryptography / trustme, one pip call per package
    (a combined install is one transaction, so one unbuildable package takes the
    whole set down -- exactly what happens on 3.13t); hypothesis fallback pin;
    and the step summary now reports coverage instead of just "passed".
  • tests/adv_util.py + 22 files -- child_timeout(), applied at exactly one
    point per call chain so the scaling cannot compound. No behaviour change at
    the default mult of 1.

Verification

Full suite green on 3.14t: 204 passed, 32 ran nothing, 0 not-passed.
Exit-code propagation through the new tee pipeline checked for
child-failure, child-success and cd-failure. Every claim above about a
package installing or a test passing was measured, not inferred.

Two findings this PR deliberately does NOT act on

1. The REALTHREAD gate is over-broad, and hides a real bug. The largest
single gap in the suite is 245 skips from one gate -- the stdlib conformance
tests that drive cooperative primitives from real OS threads, gated off as
deadlocking. Probed on 3.14t:

file with RUNLOOM_STDLIB_REALTHREAD=1
test_stdlib_queue_monkey.py 76 passed, 3 skipped -- no deadlock
test_stdlib_socket_monkey.py 79 passed, 8 skipped, 3 failed
test_stdlib_threading_monkey.py TIMEOUT at 240s in TestRunloomCondition::test_acquire

The gate is correct for threading and only for threading; it is costing
~155 conformance tests to protect against a deadlock in one file. Narrowing it
wants a Linux run and repeats first -- a deadlock in CI costs a 35-minute job
timeout.

2. Scoped IPv6 literals lose their scope id under monkey.patch(). Two of
those three socket failures are one bug. They pass on stock CPython 3.14t
and fail only under monkey:

stock  : ('fe80::1', 1234, 0, 1)
monkey : ('fe80::1%lo0', 1234, 0, 0)

src/runloom/monkey/dns.py:435, in _patched_getaddrinfo:

if aaf == socket.AF_INET6:
    sa = (a, port, 0, 0)

The address goes through verbatim with scope_id hardcoded to 0, instead of
splitting addr%iface and resolving it via if_nametoindex (it also skips
stock's inet_ntop canonicalisation -- hence 8D vs 8d in the assertion).
Not cosmetic: a link-local IPv6 address cannot be connected to without a scope
id, so connect() / create_connection() to fe80::...%eth0 works on stock
and fails under monkey. The third failure is a smaller divergence in the same
layer -- _patched_getnameinfo routes bad input through _blocking_call's
thread pool and CPython's testRefCountGetNameInfo, which catches
TypeError, sees a different type come back.

Both belong in their own change with their own tests.

A file where every collected test skips exits 0, so classify() called it
PASS and the run-level tally counted it among the passes.  _summary_line()
made it worse: it matched "passed"/"failed"/"error" but not "skipped", so
"10 skipped in 0.70s" matched nothing and the file printed with a BLANK
summary -- indistinguishable from one that genuinely ran.

Measured on CI run 35022043268, which reported green throughout:

  ubuntu/3.14   14 files executed no tests   ("236 passed, 0 skipped")
  ubuntu/3.13   16
  macOS/3.14    26

Most are legitimate -- a kqueue file SHOULD run nothing on Linux.  But
tests/test_greenlet_interop.py is in that set on every leg, and CLAUDE.md
names it the guard for the greenlet/TLBC coexistence invariant.  Nothing
installs greenlet in CI, so all 10 of its tests have always skipped, and
the board could not tell anyone.

So:
  * ALLSKIP, the missing third verdict between PASS and a failure.  Not an
    error -- OK_VERDICTS carries it alongside PASS/SKIP so the retry filter,
    the failure filter and the tally cannot drift apart.
  * -rs on the pytest line, so the REASON for each skip is in the captured
    output.
  * two unconditional end-of-run blocks: the files that executed nothing
    (with why), and a census of skipped tests by reason.  Unconditional
    because the failure mode being closed is precisely that nobody looks.
  * the tally now reads "N passed, M ran nothing, K not-passed".

"ran nothing" is the honest word: 204 passed / 32 ran nothing on 3.14t here,
where the old line said 236 passed.  The "not-passed (...)" token is
unchanged -- mac-loop.yml greps it.
…passed"

Four packages gate real tests that otherwise skip silently, with pytest
still exiting 0:

  greenlet      test_greenlet_interop.py, 10 tests.  CLAUDE.md calls that
                file the guard for the greenlet/TLBC coexistence invariant
                and it has NEVER run on CI -- nothing installed greenlet.
                Verified: 10 passed on 3.14t once present.
  hypothesis    test_chan_properties.py, 4 property tests x 200 examples
                over channel FIFO / conservation / close / select.
  cryptography  the TLS-cert arms of test_cov_ssl_edge + test_swarm_monkey.
  trustme       one TLS arm of test_mn_compat_fixes.

The old hypothesis comment said PyO3 has no free-threaded support below
3.14, so the test is simply unavailable there.  That is true only of the
NEWEST hypothesis: 6.156.1 is the first release to ship a Rust core (the
first with platform wheels rather than py3-none-any), and 6.155.7 and
earlier are pure Python.  Verified: with `hypothesis<6.156`,
test_chan_properties.py runs 4 passed on 3.13t, where it had been skipping
outright on both 3.13 legs.

One pip invocation PER PACKAGE, each best-effort.  Not tidiness: a combined
install is one transaction, so a package that cannot build takes the whole
set down with it -- exactly what happens on 3.13t, where none of these
publish a cp313t wheel.  That is the same footgun the removed comment
described for `pip install pytest hypothesis`, generalised.  Whatever does
not install is not fatal; the new skip census names what got gated off.

Also tee the suite so the step summary reports what the run COVERED.
"passed" alone was the whole problem -- this leg has been green with 14
files (26 on macOS) executing zero tests and the PR page had no way to show
it.  Exit-code propagation through the pipeline verified for child-failure,
child-success and cd-failure.
38 sites across 22 files drive a workload in a child process and, on
TimeoutExpired, pytest.skip("... timed out (shared-box contention)") rather
than fail.  That valve is deliberate: a starved child on a shared runner is
a runner problem, not a product defect.

It had a hole.  Those deadlines were plain literals -- the only deadlines in
the suite that did not move with RUNLOOM_TIMEOUT_MULT.  CI sets it to 2
precisely because its runners are slow and contended, which doubled
run_isolated.py's per-FILE ceiling and left these inner ones fixed.  So on
the very box the scaler exists for, the inner deadline trips FIRST and the
test skips instead of running slower -- inverting the scaler's whole purpose
(a slow box reads as slow, not as a failure) into a coverage hole, and an
invisible one until the census landed.

adv_util.child_timeout() scales them, applied at exactly ONE point per call
chain -- the innermost subprocess call -- so the scaling cannot compound.
Helper signature defaults and caller overrides are left alone deliberately:
they flow through the helper, which scales once.  34 call sites; thread
joins and cooperative park/recv timeouts are untouched.

No behaviour change at the default mult of 1.
@johng johng closed this Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant