Skip to content

test(interop): report why a spawned ros2 process failed - #304

Merged
YuanYuYuan merged 1 commit into
mainfrom
fix/interop-diagnostics
Aug 12, 2026
Merged

test(interop): report why a spawned ros2 process failed#304
YuanYuYuan merged 1 commit into
mainfrom
fix/interop-diagnostics

Conversation

@YuanYuYuan

@YuanYuYuan YuanYuYuan commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

The lyrical interop leg fails on every PR. The reason is currently impossible to read, because the test throws it away.

This makes the failure diagnosable. It does not fix lyrical.

Fixes #303's second half — the harness defect. The environment problem it exposes stays open there.

The defect

test_hiroz_add_two_ints_server_to_rcl_client spawns ros2 run demo_nodes_cpp add_two_ints_client with both streams sent to /dev/null, then asserts on its exit status. So a failure reports an exit code and nothing else:

Started process: RCL add_two_ints client
Stopping process group: RCL add_two_ints client
Process RCL add_two_ints client exited gracefully with status: ExitStatus(unix_wait_status(64000))

unix_wait_status(64000) is exit code 250, no signal, 23 ms after start, no output. ros2 run writes its reason to stderr — a missing executable, an unresolvable package, a library load failure all look identical here.

What this PR does

# Change Why
1 Pipe the client's stdout and stderr, and include both in the panic message on failure or timeout ros2 run reports the cause on stderr
2 Drain both pipes on reader threads (OutputCapture in tests/common/mod.rs) See below — this is the load-bearing part
3 fail-fast = false on the interop nextest profile One environment problem is not evidence about the rest of the suite

Change 2 is why change 1 is safe. The streams were nulled deliberately, and the removed comment said so: "nothing reads it, and the try_wait loop below waits on the child's exit, which a full unread pipe would block." That objection is correct. Piping without draining would let a chatty child fill its pipe buffer and block, converting a fast failure into a 30-second timeout — a worse failure mode than the one being fixed. One reader thread per stream removes the coupling; they run until EOF, which arrives when the child exits.

On change 3: before this PR the leg reported 46/154 tests run — this single failure cancelled 108 tests. A leg that stops at the first external-process problem reports far less than its green counterpart, which is the same "green means less than it looks" shape as #276. The Evidence section below measures the result.

Evidence

Both changes are demonstrated by this PR's own CI run, on the lyrical leg that #303 tracks.

Change 1 and 2 — the cause is now printed. Where the leg previously reported only ExitStatus(unix_wait_status(64000)), it now reports:

--- child stdout ---
[ros2run]: Aborted
--- child stderr ---
double free or corruption (out)

That took #303 from "the client exits 250 and we cannot tell why" to a named failure mode in one run.

Change 3 — the suite is no longer erased by it. On the same leg:

tests run passed
before 46 / 154 45
after 139 138

The single environment failure was previously cancelling 108 tests. It now reports one failure and runs the rest, which also revealed that 8 of the 9 demo_nodes interop tests pass — the failure is confined to the hiroz-service → C++-client direction. That narrowing is recorded in #303 and is what makes it actionable.

Important

No failing baseline, and none is claimed. This is a diagnostics change: no test fails without it, and no test asserts on a panic message. Its justification is the two measurements above.

Check Result
cargo check -p hiroz-tests --features ros-interop,jazzy --test demo_nodes clean
cargo fmt clean

The first compile check was run with --features ros-msgs,jazzy and passed vacuouslydemo_nodes.rs is #![cfg(feature = "ros-interop")], so that build produced an empty binary. Only the ros-interop run above verifies anything.

What this does not do

Breaking changes

None. Test-harness only.

The add_two_ints client was spawned with both streams sent to /dev/null,
so a failure could only ever be reported as an exit code. The lyrical leg
currently fails with exit 250 after 23 ms and no output, and the message
saying why is being discarded.

Pipes both streams and drains them on reader threads. Draining
concurrently is what keeps piping safe: an unread pipe fills and blocks
the child, which would turn a fast failure into a 30s timeout -- the
reason the streams were nulled in the first place.

Also sets fail-fast = false on the interop profile. One environment
problem is not evidence about the rest of the suite, and cancelling on
first failure left 108 of 154 tests unrun.
@YuanYuYuan
YuanYuYuan merged commit 6b9a966 into main Aug 12, 2026
27 of 28 checks passed
@YuanYuYuan
YuanYuYuan deleted the fix/interop-diagnostics branch August 12, 2026 10:25
YuanYuYuan added a commit that referenced this pull request Aug 14, 2026
OutputCapture::finish joins the reader threads, and they block on the open
pipes until the child exits. Both timeout branches called it while the
child was provably still running, so the panic never rendered: the test
stalled until nextest's 120s kill and printed nothing. That is the exact
failure the capture exists to prevent.

The add_two_ints branch has this defect on main already, from #304. Fixed
in both call sites.

Two smaller changes from the same review:

- The listener assertion sampled once after a fixed 1s sleep. The talker
  publishes a finite ~900ms burst, so a single sample races CI-load
  stalls. It now polls on a 15s deadline, as the reverse-direction test
  already does for the same reason.
- The action client's 30s deadline was decorative. Its server lives 10s
  and the client starts ~2s in, so a client that has not succeeded by
  then never will. Bounded at 15s and named.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lyrical interop: image ships rclcpp 32.0.0 while the repo ships 32.0.2, breaking ExecutorOptions ABI

1 participant