test(interop): report why a spawned ros2 process failed - #304
Merged
Conversation
The add_two_ints client was spawned with both streams sent to /dev/null, so a failure could only ever be reported as an exit code. The lyrical leg currently fails with exit 250 after 23 ms and no output, and the message saying why is being discarded. Pipes both streams and drains them on reader threads. Draining concurrently is what keeps piping safe: an unread pipe fills and blocks the child, which would turn a fast failure into a 30s timeout -- the reason the streams were nulled in the first place. Also sets fail-fast = false on the interop profile. One environment problem is not evidence about the rest of the suite, and cancelling on first failure left 108 of 154 tests unrun.
2 tasks
YuanYuYuan
added a commit
that referenced
this pull request
Aug 14, 2026
OutputCapture::finish joins the reader threads, and they block on the open pipes until the child exits. Both timeout branches called it while the child was provably still running, so the panic never rendered: the test stalled until nextest's 120s kill and printed nothing. That is the exact failure the capture exists to prevent. The add_two_ints branch has this defect on main already, from #304. Fixed in both call sites. Two smaller changes from the same review: - The listener assertion sampled once after a fixed 1s sleep. The talker publishes a finite ~900ms burst, so a single sample races CI-load stalls. It now polls on a 15s deadline, as the reverse-direction test already does for the same reason. - The action client's 30s deadline was decorative. Its server lives 10s and the client starts ~2s in, so a client that has not succeeded by then never will. Bounded at 15s and named.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
lyricalinterop leg fails on every PR. The reason is currently impossible to read, because the test throws it away.This makes the failure diagnosable. It does not fix
lyrical.Fixes #303's second half — the harness defect. The environment problem it exposes stays open there.
The defect
test_hiroz_add_two_ints_server_to_rcl_clientspawnsros2 run demo_nodes_cpp add_two_ints_clientwith both streams sent to/dev/null, then asserts on its exit status. So a failure reports an exit code and nothing else:unix_wait_status(64000)is exit code 250, no signal, 23 ms after start, no output.ros2 runwrites its reason to stderr — a missing executable, an unresolvable package, a library load failure all look identical here.What this PR does
stdoutandstderr, and include both in the panic message on failure or timeoutros2 runreports the cause on stderrOutputCaptureintests/common/mod.rs)fail-fast = falseon theinteropnextest profileChange 2 is why change 1 is safe. The streams were nulled deliberately, and the removed comment said so: "nothing reads it, and the try_wait loop below waits on the child's exit, which a full unread pipe would block." That objection is correct. Piping without draining would let a chatty child fill its pipe buffer and block, converting a fast failure into a 30-second timeout — a worse failure mode than the one being fixed. One reader thread per stream removes the coupling; they run until EOF, which arrives when the child exits.
On change 3: before this PR the leg reported
46/154 tests run— this single failure cancelled 108 tests. A leg that stops at the first external-process problem reports far less than its green counterpart, which is the same "green means less than it looks" shape as #276. The Evidence section below measures the result.Evidence
Both changes are demonstrated by this PR's own CI run, on the
lyricalleg that #303 tracks.Change 1 and 2 — the cause is now printed. Where the leg previously reported only
ExitStatus(unix_wait_status(64000)), it now reports:That took #303 from "the client exits 250 and we cannot tell why" to a named failure mode in one run.
Change 3 — the suite is no longer erased by it. On the same leg:
The single environment failure was previously cancelling 108 tests. It now reports one failure and runs the rest, which also revealed that 8 of the 9
demo_nodesinterop tests pass — the failure is confined to the hiroz-service → C++-client direction. That narrowing is recorded in #303 and is what makes it actionable.Important
No failing baseline, and none is claimed. This is a diagnostics change: no test fails without it, and no test asserts on a panic message. Its justification is the two measurements above.
cargo check -p hiroz-tests --features ros-interop,jazzy --test demo_nodescargo fmtThe first compile check was run with
--features ros-msgs,jazzyand passed vacuously —demo_nodes.rsis#![cfg(feature = "ros-interop")], so that build produced an empty binary. Only theros-interoprun above verifies anything.What this does not do
lyrical— but it is what made the cause findable. The captured stderr turned an exit code intodouble free or corruption (out), which led to a valgrind trace namingrclcpp::ExecutorOptions::~ExecutorOptions()freeing a stack address, and from there to the real cause: the ROS image shipsrclcpp32.0.0 while the repository serves 32.0.2, so packages installed later are built against a different ABI. Nothing to do with hiroz or zenoh — it reproduces on a stock image with the default RMW. Diagnosis in lyrical interop: image ships rclcpp 32.0.0 while the repo ships 32.0.2, breaking ExecutorOptions ABI #303; fix in ci(interop): fix the lyrical ABI skew and make interop failures diagnosable #305.Stdio::null()spawn sites indemo_nodes.rsare unchanged. None of them asserts on an exit status, so none can produce this failure mode. They can be converted if a future failure needs it.Breaking changes
None. Test-harness only.