Port upstream Symphony reliability PRs

SPEC.md

@@ -451,6 +451,9 @@ fields locally if they want stricter startup checks.
   - Default: implementation-defined.
 - `turn_sandbox_policy` (Codex `SandboxPolicy` value)
   - Default: implementation-defined.
+  - Runtime note: when the policy type is `workspaceWrite`, implementations should ensure the
+    current issue workspace remains writable even when callers add extra `writableRoots` for
+    linked-worktree metadata or similar adjunct paths.
 - `turn_timeout_ms` (integer)
   - Default: `3600000` (1 hour)
 - `read_timeout_ms` (integer)
@@ -760,6 +763,9 @@ Retry entry creation:
 
 - Cancel any existing retry timer for the same issue.
 - Store `attempt`, `identifier`, `error`, `due_at_ms`, and new timer handle.
+- Keep the issue claimed while retrying so a queued handoff cannot be dispatched twice.
+- If a claim lease exists, update it to `retrying` with retry due/backoff metadata and the last
+  seen worker/workspace details.
 
 Backoff formula:
 
@@ -784,6 +790,23 @@ Note:
 - Retry handling mainly operates on active candidates and releases claims when the issue is absent,
   rather than performing terminal cleanup itself.
 
+Claim lease behavior:
+
+- When an issue is claimed for a worker, create a visible tracker comment marker headed
+  `## Symphony Claim Lease`.
+- Persist these lease fields in runtime state and tracker-visible marker text:
+  `worker_id`, `worker_host`, `workspace_path`, `attempt`, `last_seen_at`, and
+  `lease_expires_at`.
+- The default lease TTL is derived from polling cadence: at least 60 seconds and at least three poll
+  intervals.
+- Active workers refresh the lease during poll reconciliation and when runtime/Codex activity is
+  observed.
+- Retry and blocked transitions update the lease marker with `retrying` or `blocked` state and
+  relevant error/backoff details.
+- Expired leases are recovered only when no live running or blocked worker exists for the issue.
+  Recovery logs the expiration, records an `expired` claim entry for observability, and requeues the
+  issue for retry handoff.
+
 ### 8.5 Active Run Reconciliation
 
 Reconciliation runs every tick and has two parts.
@@ -1300,6 +1323,12 @@ SHOULD return:
   - `output_tokens`
   - `total_tokens`
   - `seconds_running` (aggregate runtime seconds as of snapshot time, including active sessions)
+- `token_usage` (optional durable token summary across completed and active sessions)
+  - `input_tokens`
+  - `output_tokens`
+  - `total_tokens`
+  - `issue_count`
+  - `session_count`
 - `rate_limits` (latest coding-agent rate limit payload, if available)
 
 RECOMMENDED snapshot error modes:
@@ -1330,6 +1359,10 @@ Token accounting rules:
 - Do not treat generic `usage` maps as cumulative totals unless the event type defines them that
   way.
 - Accumulate aggregate totals in orchestrator state.
+- Implementations may also persist append-only token observations for durable per-issue
+  observability. If they do, summarize by taking the high-water cumulative totals per
+  `(issue_identifier, session_id)` and then summing those session totals; do not sum every observed
+  event.
 
 Runtime accounting:
 
@@ -1408,7 +1441,9 @@ Minimum endpoints:
       "generated_at": "2026-02-24T20:15:30Z",
       "counts": {
         "running": 2,
-        "retrying": 1
+        "retrying": 1,
+        "blocked": 0,
+        "expired": 0
       },
       "running": [
         {
@@ -1439,12 +1474,32 @@ Minimum endpoints:
           "error": "no available orchestrator slots"
         }
       ],
+      "claim_leases": [
+        {
+          "issue_id": "abc123",
+          "issue_identifier": "MT-649",
+          "state": "active",
+          "worker_id": "local:#PID<0.123.0>",
+          "workspace_path": "/tmp/symphony_workspaces/MT-649",
+          "attempt": 1,
+          "last_seen_at": "2026-02-24T20:14:59Z",
+          "lease_expires_at": "2026-02-24T20:16:30Z"
+        }
+      ],
+      "expired": [],
       "codex_totals": {
         "input_tokens": 5000,
         "output_tokens": 2400,
         "total_tokens": 7400,
         "seconds_running": 1834.2
       },
+      "token_usage": {
+        "input_tokens": 5000,
+        "output_tokens": 2400,
+        "total_tokens": 7400,
+        "issue_count": 2,
+        "session_count": 3
+      },
       "rate_limits": null
     }
     ```
@@ -1498,12 +1553,21 @@ Minimum endpoints:
         }
       ],
       "last_error": null,
-      "tracked": {}
+      "tracked": {},
+      "token_usage": {
+        "input_tokens": 1200,
+        "output_tokens": 800,
+        "total_tokens": 2000,
+        "session_count": 1
+      }
     }
     ```
 
-  - If the issue is unknown to the current in-memory state, return `404` with an error response (for
-    example `{\"error\":{\"code\":\"issue_not_found\",\"message\":\"...\"}}`).
+  - If the issue is unknown to the current in-memory state but exists in a durable token ledger, an
+    implementation may return an inactive issue payload with token usage.
+  - If the issue is unknown to both the current in-memory state and durable observability state,
+    return `404` with an error response (for example
+    `{\"error\":{\"code\":\"issue_not_found\",\"message\":\"...\"}}`).
 
 - `POST /api/v1/refresh`
   - Queues an immediate tracker poll + reconciliation cycle (best-effort trigger; implementations

elixir/README.md

@@ -31,6 +31,12 @@ issue claimed and exposes it as blocked in the runtime state, JSON API, and dash
 entries are in memory only; restarting the orchestrator clears that blocked map, so any still-active
 Linear issue can become a dispatch candidate again after restart.
 
+Claimed issues also get a Symphony claim lease marker through the tracker comment API. The lease
+records the last-seen worker id, workspace path, attempt number, last heartbeat time, and expiry.
+Active workers refresh the lease during poll and Codex activity; retry and blocked transitions
+update the same lease state. If a non-live claim lease expires, Symphony logs the recovery and
+requeues the issue without starting a duplicate worker for a still-running claim.
+
 ## How to use it
 
 1. Make sure your codebase is set up to work well with agents: see
@@ -85,6 +91,12 @@ Optional flags:
 - `--logs-root` tells Symphony to write logs under a different directory (default: `./log`)
 - `--port` also starts the Phoenix observability service (default: disabled)
 
+Symphony also writes durable Codex token usage observations to `token_usage.jsonl` next to the
+configured log file. With the default log path, this is `./log/token_usage.jsonl`; with
+`--logs-root`, it follows the same log root. The ledger stores cumulative high-water token totals
+per issue/session so completed tickets can still be inspected after the in-memory dashboard state
+has moved on.
+
 The `WORKFLOW.md` file uses YAML front matter for configuration, plus a Markdown body used as the
 Codex session prompt.
 
@@ -124,9 +136,11 @@ Notes:
   - `codex.turn_sandbox_policy` defaults to a `workspaceWrite` policy rooted at the current issue workspace
 - Supported `codex.approval_policy` values depend on the targeted Codex app-server version. In the current local Codex schema, string values include `untrusted`, `on-failure`, `on-request`, and `never`, and object-form `reject` is also supported.
 - Supported `codex.thread_sandbox` values: `read-only`, `workspace-write`, `danger-full-access`.
-- When `codex.turn_sandbox_policy` is set explicitly, Symphony passes the map through to Codex
-  unchanged. Compatibility then depends on the targeted Codex app-server version rather than local
-  Symphony validation.
+- When `codex.turn_sandbox_policy` is set explicitly, Symphony forwards the configured map to
+  Codex, but for `workspaceWrite` policies it ensures the current issue workspace stays in
+  `writableRoots` at runtime. This allows adding extra writable paths without granting access to
+  sibling workspaces by default. Compatibility for the remaining fields still depends on the
+  targeted Codex app-server version rather than local Symphony validation.
 - Workflows that run package managers or other commands that resolve external hosts should set
   `networkAccess: true` in `codex.turn_sandbox_policy`; otherwise DNS/network access may be denied
   by the Codex turn sandbox.
@@ -168,10 +182,17 @@ The observability UI now runs on a minimal Phoenix stack:
 
 - LiveView for the dashboard at `/`
 - JSON API for operational debugging under `/api/v1/*`
+- Active, retrying, blocked, and expired claim lease visibility
 - Bandit as the HTTP server
 - Phoenix dependency static assets for the LiveView client bootstrap
 - Tracker issue identifiers link to the tracker-provided URL when it uses `http` or `https`
 
+The JSON API includes durable token summaries from `token_usage.jsonl`:
+
+- `/api/v1/state` includes `token_usage` totals plus issue/session counts.
+- `/api/v1/<issue_identifier>` can return `status: "inactive"` with `token_usage` for a completed
+  or otherwise inactive issue that is no longer present in the live running/retry state.
+
 ## Project Layout
 
 - `lib/`: application code and Mix tasks

elixir/docs/token_accounting.md

@@ -285,11 +285,23 @@ That is a strong signal for Symphony:
 - use absolute totals as the main accounting surface
 - ignore last/delta values for totals
 
+## Durable Per-Issue Ledger
+
+The Elixir reference implementation persists token observations to `token_usage.jsonl` next to the
+configured log file. Each line is an append-only JSON object containing the issue identifier, Codex
+session id, source event, final/non-final marker, and cumulative input/output/total token values.
+
+The ledger is summarized by taking the maximum observed totals per `(issue_identifier, session_id)`
+and then summing those session high-water marks. This makes repeated live updates, retries, and
+final snapshots safe to append without double-counting. The ledger remains observability data only:
+it is not a billing surface and does not apply pricing or model-specific cost rules.
+
 ## Recommended Symphony Documentation Contract
 
 If Symphony documents token reporting externally, the contract should be:
 
 - Live token totals come from Codex thread-scoped cumulative usage.
+- Durable per-issue totals come from high-water cumulative totals per Codex session.
 - Incremental usage may also be emitted, but Symphony does not use it for totals.
 - Turn-completed usage is event-specific and should not be assumed to be a fresh additive increment.
 - Reporting is thread-based, and multiple turns can occur on one thread.
@@ -300,5 +312,6 @@ If Symphony documents token reporting externally, the contract should be:
 - Fallback to `info.total_token_usage`
 - Ignore `last` for totals
 - Key totals by `thread_id`
+- Persist high-water totals by `(issue_identifier, session_id)`
 - Do not classify generic `usage` by field name alone
 - Do not double-count turn-completed usage after live updates

elixir/lib/symphony_elixir/agent_runner.ex

@@ -382,14 +382,10 @@ defmodule SymphonyElixir.AgentRunner do
     Issue.stop_continue_labeled?(issue, Config.settings!().agent.stop_continue_labels)
   end
 
-  defp stop_continue_label?(_issue), do: false
-
   defp issue_routable?(%Issue{} = issue) do
     Issue.routable?(issue, Config.settings!().tracker.required_labels)
   end
 
-  defp issue_routable?(_issue), do: false
-
   defp active_issue_state?(state_name) when is_binary(state_name) do
     normalized_state = normalize_issue_state(state_name)
 

elixir/lib/symphony_elixir/config/schema.ex

@@ -353,7 +353,7 @@ defmodule SymphonyElixir.Config.Schema do
   def resolve_runtime_turn_sandbox_policy(settings, workspace \\ nil, opts \\ []) do
     case settings.codex.turn_sandbox_policy do
       %{} = policy ->
-        {:ok, policy}
+        {:ok, ensure_workspace_write_root(policy, workspace, opts)}
 
       _ ->
         workspace
@@ -579,6 +579,35 @@ defmodule SymphonyElixir.Config.Schema do
     {:error, {:unsafe_turn_sandbox_policy, {:invalid_workspace_root, workspace_root}}}
   end
 
+  defp ensure_workspace_write_root(%{"type" => "workspaceWrite"} = policy, workspace, opts) do
+    case runtime_workspace_write_root(workspace, opts) do
+      nil ->
+        policy
+
+      workspace_root ->
+        writable_roots =
+          policy
+          |> Map.get("writableRoots", [])
+          |> normalize_writable_roots()
+          |> List.insert_at(0, workspace_root)
+          |> Enum.uniq()
+
+        Map.put(policy, "writableRoots", writable_roots)
+    end
+  end
+
+  defp ensure_workspace_write_root(policy, _workspace, _opts), do: policy
+
+  defp runtime_workspace_write_root(workspace, opts)
+       when is_binary(workspace) and workspace != "" do
+    if Keyword.get(opts, :remote, false), do: workspace, else: Path.expand(workspace)
+  end
+
+  defp runtime_workspace_write_root(_workspace, _opts), do: nil
+
+  defp normalize_writable_roots(roots) when is_list(roots), do: roots
+  defp normalize_writable_roots(_roots), do: []
+
   defp default_workspace_root(workspace, _fallback) when is_binary(workspace) and workspace != "",
     do: workspace
 

elixir/lib/symphony_elixir/linear/rate_limit_budget.ex

@@ -55,7 +55,9 @@ defmodule SymphonyElixir.Linear.RateLimitBudget do
     end
   end
 
+  @spec delay_until_reset() :: :ok
   @spec delay_until_reset((non_neg_integer() -> :ok)) :: :ok
+  @spec delay_until_reset((non_neg_integer() -> :ok), non_neg_integer()) :: :ok
   def delay_until_reset(sleep_fun \\ &Process.sleep/1, fallback_ms \\ 0)
       when is_function(sleep_fun, 1) and is_integer(fallback_ms) and fallback_ms >= 0 do
     case reset_at() do