Improve concurrency handling, fix part of CI pipeline and more

.devcontainer/README.md

@@ -5,6 +5,15 @@ This directory contains the DevContainer configuration for developing FineCode i
 - `devcontainer.json`: Main configuration file for VS Code DevContainers.
 - `docker-compose.devcontainer.yml`: Compose service definition for the main workspace container.
 
+## Node.js
+
+Node.js 22 is a runtime dependency of `setup_system` handlers that install
+npm-distributed tools (e.g. pi coding agent, which rejects Node older than
+22.19.0). The base image already ships `nvm` itself (no Node version installed), so
+`setup-node.sh` installs Node 22 through that pre-existing `nvm` in
+`postCreateCommand`. Because Node is installed via nvm under
+the `vscode` user, `npm install -g` works without sudo.
+
 ## Local observability stack (opt-in)
 
 The devcontainer includes the repository-level `docker-compose.otel.yml`, but all of
@@ -22,18 +31,51 @@ Bring the stack up either way:
 
 WAL events are recorded on disk regardless of whether the stack is running, so you can
 bring it up later and ingest the history retroactively. See
-[ADR-0052](../../finecode_internal_docs/adr/0052-observability-stack-opt-in-via-compose-profile.md).
+[ADR-0052](../finecode_internal_docs/adr/0052-observability-stack-opt-in-via-compose-profile.md).
+
+## Persistent WM server
+
+`FINECODE_WM_AUTOSTART=1` is set in `.env.example`, so `postStartCommand` runs
+`start-wm-server --detach --keep-alive` on every container start, via
+`start-wm-server.sh`. The workspace stays warm across commands instead of rebuilding
+its config and runners each time. Comment the variable out to go back to a server
+per client; the script is also a no-op when the `dev_workspace` venv does not exist
+yet.
+
+Changing the variable needs the container **recreated**, not reopened — Compose
+resolves `.env` into a container's environment only when that container is created
+(the same caveat as `FINECODE_OTLP_ENDPOINT` above; see [Developing
+FineCode](../docs/guides/developing-finecode.md#local-observability-stack) for the
+rebuild commands).
+
+What comes with a keep-alive server — resident extension runners, a fixed log level,
+and why a server started lazily after a crash is not one — is described under
+[`start-wm-server`](../docs/cli.md#start-wm-server). Re-run
+`sh .devcontainer/start-wm-server.sh` to get the persistent one back.
+
+## Profiling the WM with py-spy
 
-## Optional private internal docs mount
+When the WM looks stalled, `py-spy` shows what its threads are doing without
+restarting it or changing its code. It lives in the root `dev_workspace` group, so
+it is at `.venvs/dev_workspace/bin/py-spy` after `prepare-envs --env=dev_workspace`
+(or `scripts/setup-dev-workspace.sh`).
 
-The workspace service supports an optional bind mount for private internal docs.
+Use `sudo`: Yama's `ptrace_scope=1` lets a process attach only to its own
+children, and the `vscode` user has no effective caps even with `SYS_PTRACE` in the
+container's bound set. `sudo` also resets `PATH`, so invoke the binary by path from
+the repo root.
 
-- Container target path: `/workspaces/internal-docs`
-- Host source path: `${FINECODE_INTERNAL_DOCS_PATH}`
-- Fallback when unset: `./.devcontainer/empty-internal-docs`
+```bash
+# find the pid — a dedicated WM, or the shared keep-alive one
+pgrep -f 'start-wm-server --port-file'
+pgrep -f 'start-wm-server.*--keep-alive'
 
-This means developers without private docs access can still start the devcontainer successfully.
+# one stack dump per thread
+sudo .venvs/dev_workspace/bin/py-spy dump --pid "$PID"
 
-The mount is writable from inside the container so you can edit docs directly.
+# a 90-second flame graph
+sudo .venvs/dev_workspace/bin/py-spy record --pid "$PID" --duration 90 --output wm-profile.svg
+```
 
-If you have private docs locally, set `FINECODE_INTERNAL_DOCS_PATH` in your shell or a local `.env` file before opening the devcontainer.
+`--subprocesses` is deliberately left off: it would fold every Extension Runner into
+the WM's flame graph. Profile an ER separately by pointing `--pid` at it.

.devcontainer/devcontainer.json

@@ -6,7 +6,28 @@
     ],
     "service": "workspace",
     "shutdownAction": "stopCompose",
-    "postCreateCommand": "sh .devcontainer/post-create-dev-workspace.sh && sh .devcontainer/setup-shell.sh",
+    // Node.js is required by the pi coding agent, which setup_system installs via
+    // npm. Pi's preflight rejects anything older than 22.19.0, so Debian's own
+    // nodejs package is not an option, and so is the node devcontainer feature: the
+    // base image already ships nvm itself, and running the feature on top of that
+    // hits a BuildKit bug writing into nvm's cache dir. setup-node.sh installs Node
+    // via that pre-existing nvm instead, at container-create time. See
+    // setup-node.sh for the full explanation.
+    // remoteEnv (not containerEnv): containerEnv is baked into the compose
+    // service's own environment, including its `sleep infinity` entrypoint process —
+    // and "${containerEnv:PATH}" does not self-resolve there, so it overwrote PATH
+    // with the literal unexpanded string and broke everything, including `sleep`.
+    // remoteEnv only applies to devcontainer-managed processes (postCreateCommand,
+    // terminals), where the substitution does work.
+    "remoteEnv": {
+        "NVM_DIR": "/usr/local/share/nvm",
+        "PATH": "/usr/local/share/nvm/current/bin:${containerEnv:PATH}"
+    },
+    "postCreateCommand": "sh .devcontainer/setup-node.sh && sh .devcontainer/post-create-dev-workspace.sh && sh .devcontainer/setup-shell.sh",
+    // postStart, not postCreate: a stopped container loses its WM server, so this
+    // has to run again on every start. The venv it needs may not exist yet on the
+    // first create; the script exits cleanly in that case.
+    "postStartCommand": "sh .devcontainer/start-wm-server.sh",
     "customizations": {
         "vscode": {
             "extensions": [

.devcontainer/docker-compose.devcontainer.yml

@@ -3,6 +3,10 @@ services:
     image: mcr.microsoft.com/devcontainers/python:3.14
     init: true
     user: vscode
+    # py-spy needs ptrace/process_vm_readv; Docker's default seccomp profile
+    # allows them only with SYS_PTRACE. The devcontainer is dev-only; CI does
+    # not use this compose file.
+    cap_add: [SYS_PTRACE]
     working_dir: /workspaces/finecode
     command: sleep infinity
     environment:
@@ -11,6 +15,9 @@ services:
       # both COMPOSE_PROFILES=otel and FINECODE_OTLP_ENDPOINT in .env — the ready
       # collector URL is provided there, so developers never compose it themselves.
       - FINECODE_OTLP_ENDPOINT=${FINECODE_OTLP_ENDPOINT:-}
+      # Read only by .devcontainer/start-wm-server.sh; passed through here so the
+      # value set in .env is visible to processes inside the container.
+      - FINECODE_WM_AUTOSTART=${FINECODE_WM_AUTOSTART:-}
       # uv's default cache (~/.cache/uv) lives on the container's overlay
       # filesystem, a different device than this bind-mounted workspace. uv
       # dedupes installs by hardlinking from its cache into each venv, but
@@ -21,9 +28,6 @@ services:
     volumes:
       - .:/workspaces/finecode:cached
       - ./.claude:/home/vscode/.claude:cached
-      - type: bind
-        source: ${DEVCONTAINER_INTERNAL_DOCS_PATH:-./.devcontainer/empty-internal-docs}
-        target: /workspaces/finecode_internal_docs
     networks:
       - finecode-net
     depends_on:

.devcontainer/setup-node.sh

@@ -0,0 +1,24 @@
+#!/bin/sh
+# Installs Node via the nvm already baked into the base image.
+#
+# We do NOT use the ghcr.io/devcontainers/features/node feature for this:
+# the base image already ships nvm itself (no Node version installed yet)
+# at /usr/local/share/nvm, owned by vscode:nvm. Running the node feature on
+# top of that hits its "NVM already installed" code path, which skips the
+# ownership fixup it only does for a freshly created nvm dir and calls
+# `nvm install` directly. That specific command works fine under a normal
+# `docker run`, but fails with "Permission denied" writing into
+# /usr/local/share/nvm/.cache when run inside a BuildKit RUN layer, which
+# does not reproduce ownership from the base image's read-only layer
+# correctly on copy-up. Running nvm install here, in postCreateCommand
+# (a normal container process, not a build layer) sidesteps the bug.
+set -eu
+
+export NVM_DIR="/usr/local/share/nvm"
+export NVM_SYMLINK_CURRENT=true
+# shellcheck source=/dev/null
+. "$NVM_DIR/nvm.sh"
+
+umask 0002
+nvm install 22
+nvm alias default 22

.devcontainer/start-wm-server.sh

@@ -0,0 +1,29 @@
+#!/bin/sh
+# Start a persistent shared FineCode WM server at container start, so the
+# workspace stays warm for the life of the container. What that costs is
+# documented in docs/cli.md, "Autostarting a persistent server".
+#
+# FineCode itself never reads FINECODE_WM_AUTOSTART — this script is its only
+# consumer, and it turns it into an explicit `--keep-alive` on the server it
+# starts, because keep-alive must never be ambient.
+set -eu
+
+if [ -z "${FINECODE_WM_AUTOSTART:-}" ] || [ "${FINECODE_WM_AUTOSTART}" = "0" ]; then
+    exit 0
+fi
+
+VENV_PYTHON=".venvs/dev_workspace/bin/python"
+if [ ! -x "$VENV_PYTHON" ]; then
+    echo "FineCode WM autostart: $VENV_PYTHON not found, skipping." >&2
+    exit 0
+fi
+
+# Never fatal: the workspace is fully usable without it, because every client
+# starts a server on demand anyway. Failing container start over a warm cache
+# would be a worse outcome than the cold start it is trying to avoid.
+if "$VENV_PYTHON" -m finecode start-wm-server --detach --keep-alive; then
+    echo "FineCode WM autostart: shared server running."
+else
+    echo "FineCode WM autostart: no shared server reachable yet; it may still be" \
+        "starting, and clients attach to it when it is." >&2
+fi

.env.example

@@ -1,10 +1,6 @@
 # Local developer environment variables.
 # Copy values you need into .env for your machine.
 
-# Optional: absolute or repo-relative host path to private internal docs.
-# When unset, devcontainer falls back to ./.devcontainer/empty-internal-docs.
-DEVCONTAINER_INTERNAL_DOCS_PATH=
-
 # Optional: local observability stack (grafana/otel-lgtm, bundling the OTel
 # Collector, Tempo, Prometheus, Loki, and Grafana, plus a standalone Jaeger).
 # These services are gated behind the "otel" Compose profile and stay DOWN by
@@ -20,3 +16,9 @@ DEVCONTAINER_INTERNAL_DOCS_PATH=
 # use it to start/stop the collector while the endpoint stays armed.
 # COMPOSE_PROFILES=otel
 # FINECODE_OTLP_ENDPOINT=http://otel-lgtm:4317
+
+# Start a persistent shared WM server when the devcontainer starts, so config and
+# extension runners survive between commands. Comment out to get a server per
+# client instead. What comes with it: docs/cli.md, "Autostarting a persistent
+# server".
+FINECODE_WM_AUTOSTART=1

.github/ci/finecode-user.ci.toml

@@ -0,0 +1,12 @@
+# CI counterpart to a developer's gitignored finecode-user.toml.
+#
+# The `.ci` infix is load-bearing: .gitignore ignores the unanchored name
+# `finecode-user.toml` at any depth, so a file here named exactly
+# `finecode-user.toml` could never be committed. This name is not ignored, which
+# is what lets the public repo track the CI copy while the developer copy stays
+# out of git. Do not "tidy" the infix away.
+
+presets = [{ source = "fine_knowledge" }]
+
+[dependency-groups]
+dev_workspace = ["fine_knowledge~=0.1.0a0"]