Snapshot spawned server processes with psutil on every platform

The startup-timeout snapshot read /proc directly, so it returned an
empty string off Linux and a failed start on Windows or macOS said
nothing about what the spawned server was doing at the deadline.

- Replace describe_process_group with describe_spawned_processes,
  built on psutil. On POSIX the server is spawned with
  start_new_session=True, so its pid is its process group and every
  member is listed, including children reparented after their parent
  exited. Windows has no process group, so the tree rooted at the
  spawned pid is listed by parent pid, falling back to processes whose
  ppid is the root when it already died. Windows lines omit state=
  (psutil reports `running` for nearly every process) and add ppid and
  thread count.
- An unavailable snapshot now says why ("process snapshot unavailable:
  <reason>") instead of a fixed string, and never raises so it cannot
  mask the start error it explains.
- Declare psutil as a finecode_jsonrpc dependency; it was previously
  only a dev/test dependency of the root project.
- Cover the per-platform behaviour in test_proc_snapshot and the
  ServerFailedToStart message in test_startup_timeline.
- Update wm-server-internals.md; the Job Object replacement that would
  make Windows membership a spawn-side record is tracked in issue 43.

Author: Aksem

docs/guides/wm-server-internals.md

@@ -468,9 +468,14 @@ WARNING by `_start_extension_runner_process`, carrying `spawn_to_output_ms`, `sp
 and `spawn_to_connected_ms` plus the host's memory/IO pressure; a faster start is logged at DEBUG
 with the same fields. When the port handshake itself times out, the `ServerFailedToStart` message
 names the timeline (when it was spawned, whether the server produced any output, when its port
-line arrived) and a snapshot of the spawned process group at the deadline
+line arrived) and a snapshot of the spawned server's processes at the deadline
 (`finecode_jsonrpc._proc_snapshot`), followed by the stdout/stderr tails; the failing-start log
-line also carries `spawned_ago_ms` and host pressure. A start that reaches RUNNING logs
+line also carries `spawned_ago_ms` and host pressure. The snapshot is per-platform: on POSIX the
+server is spawned with `start_new_session=True`, so its pid is its process group and every member
+is listed (including children reparented after their parent exited); on Windows there is no process
+group, so the process tree rooted at the spawned pid is listed by parent pid, with no `state=`
+(psutil reports `running` for nearly every Windows process — see issue 43 for the Job Object
+replacement that would make the membership a spawn-side record). A start that reaches RUNNING logs
 `Runner <id> ready: connected_to_running_ms=… spawn_to_running_ms=…` (DEBUG, WARNING at
 ≥ 10 s total); a start abandoned before RUNNING logs `Runner <id> abandoned:`
 `connected_to_abandon_ms=…` when its channel had connected. `initialize` / `updateConfig` failures

finecode_jsonrpc/pyproject.toml

@@ -10,6 +10,7 @@ dependencies = [
   "culsans==0.11.*",
   "finecode_extension_api~=0.5.0a0",
   "loguru==0.7.*",
+  "psutil==7.2.*",
   "typing-extensions (>=4.12.2,<5.0.0)",
 ]
 

finecode_jsonrpc/src/finecode_jsonrpc/_proc_snapshot.py

@@ -1,89 +1,162 @@
+"""Per-platform diagnostic snapshot of a spawned server's processes.
+
+A start failure needs to say what the spawned server was doing at the
+deadline: its processes listed, "no live process" if it already exited, or
+"snapshot unavailable" with a reason. Never raises: a snapshot failure must
+not replace the start error it is explaining.
+"""
+
 from __future__ import annotations
 
 import os
 import sys
 import typing
 
-__all__ = ["describe_process_group"]
+import psutil
 
+__all__ = ["describe_spawned_processes"]
 
-class _ProcStat(typing.NamedTuple):
-    comm: str
-    state: str
-    pgrp: int
-    utime: int
-    stime: int
+# ``os.getpgid`` is POSIX-only in typeshed, but the name must resolve on
+# Windows too, where CI-win's type check runs with the win32 platform. Probed
+# 2026-09-24: a direct ``os.getpgid`` → ``missing-attribute`` under
+# ``--python-platform win32``, so it is looked up dynamically at module level.
+_getpgid = getattr(os, "getpgid", None)
 
 
-def _parse_stat(text: str) -> _ProcStat:
-    """Parse ``/proc/<pid>/stat``.
+def describe_spawned_processes(
+    root_pid: int, limit: int = 10, platform: str = sys.platform
+) -> str:
+    """One diagnostic line per process the spawned server consists of.
 
-    ``comm`` is wrapped in parentheses and may itself contain spaces or
-    parentheses, so the fields after it are read from the *last* ``)`` — a
-    naive ``split()`` would misalign on such a name.
+    POSIX: the server is spawned with ``start_new_session=True``, so its pid is
+    its process group; every member is listed, including children reparented
+    after their parent exited. Windows has no process group: the tree rooted at
+    *root_pid*, by parent pid (see issue 43 for the Job Object replacement).
+    Returns the lines, ``"no live process in …"``, or
+    ``"process snapshot unavailable: <reason>"``. Never raises: a snapshot
+    failure must not replace the start error it is explaining.
     """
-    comm = text[text.index("(") + 1 : text.rfind(")")]
-    fields = text[text.rfind(")") + 2 :].split()
-    return _ProcStat(
-        comm=comm,
-        state=fields[0],
-        pgrp=int(fields[2]),
-        utime=int(fields[11]),
-        stime=int(fields[12]),
-    )
+    by_group = platform != "win32"
+    with_swap = platform.startswith("linux")
+    with_state = platform != "win32"
+    with_tree_fields = platform == "win32"
 
+    try:
+        members = _members(root_pid, by_group)
+        if not members:
+            if by_group:
+                return f"no live process in process group {root_pid}"
+            return f"no live process in the process tree of {root_pid}"
 
-def _read_status_memory(pid: int) -> tuple[int, int]:
-    """(VmRSS, VmSwap) in kB; zeros where the kernel does not report them."""
-    rss_kb = 0
-    swap_kb = 0
-    with open(f"/proc/{pid}/status") as status_file:
-        for line in status_file:
-            if line.startswith("VmRSS:"):
-                rss_kb = int(line.split()[1])
-            elif line.startswith("VmSwap:"):
-                swap_kb = int(line.split()[1])
-    return rss_kb, swap_kb
+        lines: list[str] = []
+        for pid in sorted(members):
+            process = members[pid]
+            with process.oneshot():
+                try:
+                    name = process.name()
+                except (psutil.NoSuchProcess, psutil.ZombieProcess):
+                    # It vanished mid-read — it is not part of the snapshot,
+                    # and not an error.
+                    continue
+                fields = [f"{pid} {name}"]
+                if with_state:
+                    fields.append(f"state={_read_status(process)}")
+                if with_tree_fields:
+                    fields.append(f"ppid={_read_ppid(process)}")
+                fields.append(f"cpu={_read_cpu_seconds(process)}")
+                fields.append(f"rss={_read_rss(process)}")
+                if with_swap:
+                    fields.append(f"swap={_read_swap(process)}")
+                if with_tree_fields:
+                    fields.append(f"threads={_read_threads(process)}")
+                lines.append(" ".join(fields))
+            if len(lines) >= limit:
+                break
+        return "\n".join(lines)
+    except Exception as exception:  # noqa: BLE001
+        return f"process snapshot unavailable: {type(exception).__name__}: {exception}"
 
 
-def describe_process_group(pgid: int, limit: int = 10) -> str:
-    """One diagnostic line per process in process group *pgid*, or ``""`` off Linux.
+def _members(root_pid: int, by_group: bool) -> dict[int, psutil.Process]:
+    """The processes the spawned server consists of, keyed by pid.
 
-    A start failure needs to say what the spawned process group was doing at the
-    deadline: a process group with no members means the shell already exited, one
-    running at full CPU is working, one parked in ``D`` state is blocked on I/O.
-    Never raises: a snapshot failure must not replace the start error it is
-    explaining.
-    """
-    if not sys.platform.startswith("linux"):
-        return ""
+    Tree (Windows): ``Process(root_pid)`` plus ``.children(recursive=True)``
+    — psutil drops a child whose creation time is earlier than the parent's,
+    which is the pid-reuse guard. If the root is dead, every process whose
+    parent pid equals *root_pid*, plus their descendants, is added instead:
+    Windows never rewrites ppid, so a child orphaned by its server's exit is
+    still found this way.
 
-    try:
-        clock_ticks = os.sysconf("SC_CLK_TCK")
-        lines: list[str] = []
-        for entry in sorted(
-            os.listdir("/proc"), key=lambda name: int(name) if name.isdigit() else 0
-        ):
-            if not entry.isdigit():
-                continue
-            pid = int(entry)
+    Group (POSIX): every process whose process group id equals *root_pid*,
+    found through ``os.getpgid`` even when a member was reparented after its
+    parent exited.
+    """
+    members: dict[int, psutil.Process] = {}
+    if not by_group:
+        try:
+            members[root_pid] = psutil.Process(root_pid)
+        except psutil.NoSuchProcess:
+            for entry in psutil.process_iter(["ppid"]):
+                if entry.info["ppid"] == root_pid:
+                    members[entry.pid] = entry
+        for pid in list(members):
+            _add_children(members, pid)
+    elif _getpgid is not None:
+        for entry in psutil.process_iter():
             try:
-                with open(f"/proc/{pid}/stat") as stat_file:
-                    stat = _parse_stat(stat_file.read())
-                if stat.pgrp != pgid:
-                    continue
-                rss_kb, swap_kb = _read_status_memory(pid)
-            except OSError:
-                # The process vanished while we were reading it — it is not
-                # part of the snapshot, and not an error.
+                if _getpgid(entry.pid) == root_pid:
+                    members[entry.pid] = entry
+            except (ProcessLookupError, PermissionError):
+                # Vanished or owned by another user — not part of the group.
                 continue
-            cpu_sec = (stat.utime + stat.stime) / clock_ticks
-            lines.append(
-                f"{pid} {stat.comm} state={stat.state}"
-                f" cpu={cpu_sec:.2f}s rss={rss_kb // 1024}MB swap={swap_kb // 1024}MB"
-            )
-            if len(lines) >= limit:
-                break
-        return "\n".join(lines)
-    except Exception:  # noqa: BLE001
-        return "process group snapshot unavailable"
+    return members
+
+
+def _add_children(members: dict[int, psutil.Process], parent_pid: int) -> None:
+    try:
+        children = members[parent_pid].children(recursive=True)
+    except psutil.NoSuchProcess:
+        return
+    for child in children:
+        members.setdefault(child.pid, child)
+
+
+def _read_status(process: psutil.Process) -> str:
+    return _read(lambda: str(process.status()))
+
+
+def _read_ppid(process: psutil.Process) -> str:
+    return _read(lambda: str(process.ppid()))
+
+
+def _read_threads(process: psutil.Process) -> str:
+    return _read(lambda: str(process.num_threads()))
+
+
+def _read_cpu_seconds(process: psutil.Process) -> str:
+    return _read(
+        lambda: f"{process.cpu_times().user + process.cpu_times().system:.2f}s"
+    )
+
+
+def _read_rss(process: psutil.Process) -> str:
+    return _read(lambda: f"{process.memory_info().rss // 2**20}MB")
+
+
+def _read_swap(process: psutil.Process) -> str:
+    return _read(lambda: f"{process.memory_full_info().swap // 2**20}MB")
+
+
+def _read(read: typing.Callable[[], object]) -> str:
+    """One per-process field, stringified; ``?`` when the read is denied.
+
+    ``AccessDenied`` is expected for processes owned by another user; a process
+    that vanished mid-read is reported the same way rather than dropped for the
+    rest of its line.
+    """
+    try:
+        return str(read())
+    except psutil.AccessDenied:
+        return "?"
+    except (psutil.NoSuchProcess, psutil.ZombieProcess):
+        return "?"

finecode_jsonrpc/src/finecode_jsonrpc/client.py

@@ -32,7 +32,7 @@
 from finecode_jsonrpc import _io_thread, _spawn, error_codes
 from finecode_jsonrpc._converter import converter as _converter
 from finecode_jsonrpc._loop_event import LoopAwareEvent
-from finecode_jsonrpc._proc_snapshot import describe_process_group
+from finecode_jsonrpc._proc_snapshot import describe_spawned_processes
 from finecode_jsonrpc.tracing import ITracingHooks
 
 
@@ -1016,15 +1016,15 @@ async def _connect_to_server_io(self, timeout: float | None) -> None:
                 for task in self._async_tasks_in_io_thread:
                     task.cancel()
 
-                process_group_snapshot = (
-                    describe_process_group(self.pid)
+                process_snapshot = (
+                    describe_spawned_processes(self.pid)
                     if self.pid is not None
-                    else "process group snapshot unavailable"
+                    else "process snapshot unavailable: server pid unknown"
                 )
                 raise ServerFailedToStart(
                     f"Didn't get port in {timeout} seconds"
                     f"\nStartup timeline: {self.startup_timeline.describe(time.monotonic())}"
-                    f"\nServer process group at timeout:\n{process_group_snapshot}"
+                    f"\nServer processes at timeout:\n{process_snapshot}"
                     f"{self._stdout_tail()}{self._stderr_tail()}"
                 ) from exception