Fix: teleop freeze from portal signal-pipe leak in minimum_gello

[ronaldnetawat]

Problem

Leader-follower teleop with examples/minimum_gello/minimum_gello.py freezes reliably after ~2–3 minutes of synced driving: the follower stops tracking and holds its last commanded pose rigidly, the teaching-handle sync button appears dead, and neither process logs any error — both keep printing healthy rate reports. Idle (unsynced) sessions never freeze, even overnight.

Root cause

portal's client (3.7.3) wakes its socket thread by writing one byte to an internal signal pipe per request (client_socket.py send()), but the socket loop drains that pipe at most one byte per send-queue-emptying — the if not writing: guard skips the drain while messages are queued, and the post-drain writing = True persists until the next queue-emptying.

_run_leader_io_loop issues command_joint_pos fire-and-forget and immediately follows it with get_joint_pos. Both requests land in one send burst: two signal bytes written, one drained — a leak of one pipe byte per loop cycle while synced. At a few hundred cycles/sec the 64 KiB pipe fills in ~2–3 minutes, after which os.write on the full blocking pipe blocks forever inside send(). The io loop dies silently; the leader's control worker keeps running at full rate, so no commands reach the follower (rigid at last pose) and desync toggles never get forwarded (button "dead"). Unsynced sessions issue one request per cycle, which happens to be exactly drain-balanced — which is why the freeze only appears after enabling sync.

py-spy dumps of two independent freezes show the main thread stuck at the same line (once via get_joint_pos, once via command_joint_pos):

Thread (idle): "MainThread"
    send (portal/client_socket.py:81)      # os.write(self.set_signal, bytes(1)) on full pipe
    call (portal/client.py:88)
    command_joint_pos (minimum_gello.py:50)
    _run_leader_io_loop (minimum_gello.py:395)

Fix

• Pace _run_leader_io_loop and _rpc_polling_worker with _WORKER_LOOP_PERIOD_S, matching every other loop in the file (unpaced they run ~7 kHz of RPC against a ~270 Hz robot).
• Call .result() on ClientRobot.command_joint_pos / command_joint_state. The wait is load-bearing, not just error propagation: completing each request before issuing the next keeps every request in its own send burst, so signal-pipe writes and drains stay exactly 1:1 balanced indefinitely. Comments in the code document the mechanism.

A deeper fix (draining all pending signal bytes) belongs upstream in portal; this change makes the example robust with portal as-is.

Verification

Real hardware — YAM leader (yam_teaching_handle) + YAM follower (linear_4310), separate CANable buses, --bilateral-kp 0.2: 10+ minutes of continuous synced teleop with zero io stalls (857 consecutive healthy 1 s rate reports), versus a reliable silent freeze within ~3 minutes before the change. The io loop runs at ~365–415 Hz after the fix.