Skip to content

fix(init): redial the substrate vsock instead of going silent forever#15

Merged
jaredLunde merged 1 commit into
mainfrom
fix/substrate-vsock-reconnect
Jul 14, 2026
Merged

fix(init): redial the substrate vsock instead of going silent forever#15
jaredLunde merged 1 commit into
mainfrom
fix/substrate-vsock-reconnect

Conversation

@jaredLunde

Copy link
Copy Markdown
Contributor

The bug

The substrate vsock connection is how the host knows this guest exists — instd pings every ~10s and marks the VM Degraded (guest disconnected) after 30s of silence.

The thread owning it exited on EOF and never redialled. So any instd restart (i.e. every deploy:compute:local) left the VM running but permanently invisible to the host: no ready, no heartbeat replies, no PSI reports, no workload logs — and nothing to ever restore them.

Found while fixing the sibling bug in the Beyond repo, where guest-runtime primitives (vps/oci/service/echo/canary) powered themselves off on an instd restart. This repo always got that part right — PID 1 here is supervise()'s poll loop over vsock listeners, not a client, so a dropped outbound connection can't end it ("this thread never powers the VM off"). The VM survived. It just never came back.

The fix

keep_alive now reports why it returned:

  • Disconnected → redial with capped backoff, re-sending Ready { reconnect: true } — a protocol field that already existed and was hardcoded false.
  • Shutdown → stop for good. That is not a disconnect, and retrying through it would fight the supervise loop's poweroff.

A first connect failure stays soft-fail and exits the thread: that means there's no substrate here at all (Docker tests, no AF_VSOCK), so there's nothing to redial. Having connected once proves a substrate exists, and a host that went away is a host that's coming back.

The log sink moves outside the reconnect loop — it owns a unix listener, and rebinding per reconnect would unlink the socket under the previous accept task and race it. Its receiver carries across a host bounce so the supervisor keeps relaying into the same channel.

Tests

Two new tests pin the invariant that caused this, driving keep_alive over an in-memory duplex:

  • eof_is_a_disconnect_not_a_shutdown
  • explicit_shutdown_frame_stops_for_good

plus redial_marks_the_ready_frame_as_a_reconnect. Full suite: 36 passed. Clippy clean (the 2 collapsible_if warnings in bootsetup.rs are pre-existing and untouched).

🤖 Generated with Claude Code

The substrate connection IS how the host knows this guest exists: instd pings
every ~10s and marks the VM `Degraded (guest disconnected)` after 30s of silence.
The thread that owns it exited on EOF and never redialled, so any instd restart
(every deploy:compute:local) left the VM running but permanently invisible: no
ready, no heartbeat replies, no PSI reports, no workload logs, and nothing to
ever restore them.

The VM itself always survived -- unlike the guest-runtime primitives, PID 1 here
is supervise()'s poll loop over vsock LISTENERS, not a client, so a dropped
outbound connection can't end it. This repo already got that part right ("this
thread never powers the VM off"). It just never came back.

keep_alive now reports WHY it returned. Disconnected -> redial with capped
backoff, re-sending Ready{reconnect:true} (a protocol field that already existed
and was hardcoded false). Shutdown -> stop for good; that is not a disconnect and
retrying through it would fight the supervise loop's poweroff.

A FIRST connect failure stays soft-fail and exits the thread: it means there is
no substrate here at all (Docker tests, no AF_VSOCK) and so nothing to redial.
Having connected once proves a substrate exists, and a host that went away is a
host that is coming back.

The log sink moves outside the reconnect loop -- it owns a unix listener, and
rebinding per reconnect would unlink the socket under the previous accept task
and race it. Its receiver carries across a host bounce, so the supervisor keeps
relaying into the same channel.

Tests pin the invariant that caused this: EOF is a disconnect, not a shutdown
(and an explicit Shutdown frame still stops for good). Both drive keep_alive over
an in-memory duplex.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jaredLunde jaredLunde merged commit b89af74 into main Jul 14, 2026
1 check passed
@jaredLunde jaredLunde deleted the fix/substrate-vsock-reconnect branch July 14, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant