fix(init): redial the substrate vsock instead of going silent forever#15
Merged
Conversation
The substrate connection IS how the host knows this guest exists: instd pings
every ~10s and marks the VM `Degraded (guest disconnected)` after 30s of silence.
The thread that owns it exited on EOF and never redialled, so any instd restart
(every deploy:compute:local) left the VM running but permanently invisible: no
ready, no heartbeat replies, no PSI reports, no workload logs, and nothing to
ever restore them.
The VM itself always survived -- unlike the guest-runtime primitives, PID 1 here
is supervise()'s poll loop over vsock LISTENERS, not a client, so a dropped
outbound connection can't end it. This repo already got that part right ("this
thread never powers the VM off"). It just never came back.
keep_alive now reports WHY it returned. Disconnected -> redial with capped
backoff, re-sending Ready{reconnect:true} (a protocol field that already existed
and was hardcoded false). Shutdown -> stop for good; that is not a disconnect and
retrying through it would fight the supervise loop's poweroff.
A FIRST connect failure stays soft-fail and exits the thread: it means there is
no substrate here at all (Docker tests, no AF_VSOCK) and so nothing to redial.
Having connected once proves a substrate exists, and a host that went away is a
host that is coming back.
The log sink moves outside the reconnect loop -- it owns a unix listener, and
rebinding per reconnect would unlink the socket under the previous accept task
and race it. Its receiver carries across a host bounce, so the supervisor keeps
relaying into the same channel.
Tests pin the invariant that caused this: EOF is a disconnect, not a shutdown
(and an explicit Shutdown frame still stops for good). Both drive keep_alive over
an in-memory duplex.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The substrate vsock connection is how the host knows this guest exists — instd pings every ~10s and marks the VM
Degraded (guest disconnected)after 30s of silence.The thread owning it exited on EOF and never redialled. So any instd restart (i.e. every
deploy:compute:local) left the VM running but permanently invisible to the host: no ready, no heartbeat replies, no PSI reports, no workload logs — and nothing to ever restore them.Found while fixing the sibling bug in the Beyond repo, where
guest-runtimeprimitives (vps/oci/service/echo/canary) powered themselves off on an instd restart. This repo always got that part right — PID 1 here issupervise()'s poll loop over vsock listeners, not a client, so a dropped outbound connection can't end it ("this thread never powers the VM off"). The VM survived. It just never came back.The fix
keep_alivenow reports why it returned:Disconnected→ redial with capped backoff, re-sendingReady { reconnect: true }— a protocol field that already existed and was hardcodedfalse.Shutdown→ stop for good. That is not a disconnect, and retrying through it would fight the supervise loop's poweroff.A first connect failure stays soft-fail and exits the thread: that means there's no substrate here at all (Docker tests, no AF_VSOCK), so there's nothing to redial. Having connected once proves a substrate exists, and a host that went away is a host that's coming back.
The log sink moves outside the reconnect loop — it owns a unix listener, and rebinding per reconnect would unlink the socket under the previous accept task and race it. Its receiver carries across a host bounce so the supervisor keeps relaying into the same channel.
Tests
Two new tests pin the invariant that caused this, driving
keep_aliveover an in-memory duplex:eof_is_a_disconnect_not_a_shutdownexplicit_shutdown_frame_stops_for_goodplus
redial_marks_the_ready_frame_as_a_reconnect. Full suite: 36 passed. Clippy clean (the 2collapsible_ifwarnings inbootsetup.rsare pre-existing and untouched).🤖 Generated with Claude Code