[dawsontoth]

⊙ Problem

Deployment tracking (#641) left gaps. A rolling deploy's record, and harper deploy restart=rolling, reported success before other nodes took the release. Nothing reported deploy or restart times, a node's kept releases, or a deploy's origin. Upgraded installs missed newer hdb_deployment attributes. A truncated payload could report success. Failed payloads were kept forever. An internal event leaked into deploy history.

🤔 Does all of this belong in one PR?

Asked for as one PR, one commit per issue, so each reads and reverts on its own. The commits share components/deploymentRecorder.ts and components/DESIGN.md, which is why they were not split.

💡 Solution

• A rolling deploy's record stays rolling_out until every node took the release, then ends success or partial naming the nodes that did not. harper deploy waits on that record.
• get_components gives each application deployment_id, deployed_at and restarted_at; new list_releases lists each node's releases.
• Deployment rows record source, attested by OIDC or declared by the CLI.
• A deploy warns before going live when a declared jsResource/graphqlSchema entry point is missing.
• Failed payloads expire; upgrades register new system-table attributes.

🤔 Should a missing entry point fail the deploy (#1719)?

It warns, so #1719 stays open. Failing broke a tested workflow: redeploying to delete an application's only resource file, whose config still names it (redeploy-restart-flag-deletion). Making it fatal, ending that workflow, is one line.

⚖️ Alternatives

• Track per-application deployment and restart timestamps #564 from configuration history — a payload deploy changes no configuration, so the most common deploy would be missed; hdb_deployment already records every deploy.
• harper deploy restart=rolling exits 0 before the other nodes have taken the release #3058 by polling get_job — hdb_job does not replicate, so a poll behind a load balancer can land on a node that never had the job.
• Report which deployment is live on each node, and which releases each node keeps #3059 as more get_components fields — sizing each kept tree walks it on every get_components poll; list_releases pays that only when asked.
• Discard the rest of a multipart upload whose route's pipe loses its destination after the response #3096 on pause or removeListener alone — backpressure and Node 24.0/22.18 each miss one; every lost pipe calls the part's unpipe().

🔧 Changes

Left for later: #1719's fatal check (above); an uncertified restart=rolling still records success before its peers restart, and the CLI does not wait on it; a payload's install fingerprint still has no source (#3061); a failed payload whose node left the cluster is never expired; a release kept before 5.4 reports staged. Docs need a companion PR in HarperFast/documentation.

🤔 Should the rollout's state live on the deployment row?

Yes: get_deployment and list_deployments already read the row, and it replicates. The cost is write ordering, handled by the job's wait and spacing and a node-scoped lock on the row. Separate per-peer records would avoid contention but give readers two places to look.

🤔 Should harper deploy restart=rolling wait by default?

Yes: #3058 asks that CI not pass before every node took the release. A CI step that exited at the origin's swap now waits up to ten minutes per node and can fail; wait=false keeps the old exit.

🤔 Should restarted_at count a load that failed?

Yes: it says when this node's workers last loaded the application; status says whether that worked. Renaming it later would break callers.

🤔 Should list_deployments { sha } match declared sources?

Yes: each row's source.via says whether a CI exchange attested it. Matching only attested sources would hide every deploy from a laptop.

Per issue: what changed, with links

#3062, the recorder's internal event. finish() no longer emits _recorder_finished; the registry of active deploys holds a promise it settles after the terminal write. The registry entry is removed only after that write. The get_deployment tail awaits it, or its client's disconnect, and re-reads the final row.

#2659, attributes on upgraded installs. registerSystemSchemaAttributes adds every attribute systemSchema.json declares that an existing system table lacks, indexed like a fresh install. The 5.4.0 directive runs it (registered). mount_hdb and the 5.1, 5.2 and 5.3 directives now pass table() a copy of the schema's attribute list, since table() kept the bundled array as the live list. The rule is in dataLayer/DESIGN.md; systemSchema.json gains restart_job_id and source.

#564, deploy and restart times. getComponents reads the live marker (liveDeploymentIdOf) and finds the record that made it live by key and by the activated_from index, not a scan of the history; a catalog that cannot search that index yet keeps the record found by key. A tree with no marker takes the newest row of each released status, so no run of failed attempts hides it. restarted_at comes from loadedAt, set when a thread finishes loading, kept across reports, aggregated across threads and sub-components (types).

#3060, peer outcomes on the record. The origin finishes a rolling deploy as rolling_out with restart_job_id. The activation job records each peer and ends the record success or partial. The origin's own write keeps what the job already stored, and a boot ends records whose job was interrupted (at boot). ⚠️ Look hardest: the job waits for the origin's rolling_out write before its first; the last peer's outcome, and any whose own write failed, rides the final write and every write waits a second after the previous one, so writes to the row never land as a burst (#1170). Each read-merge-write, the origin's and the job's, holds the row's node-scoped lock, since a conflict retry replays a put without rerunning its merge (LMDB has none). The final write retries, then fails the job with every outcome in its record, which the boot pass reads back. partial is terminal. The job loads the cluster's node list itself: a job worker thread starts with an empty server.nodes, so until this branch a certified rolling deploy activated on no peer of a real cluster, on main too. Only core's replication stand-in means no peers, and a job that fails before it can finish the record ends it partial (abandonRollout).

#3058, the CLI waits. awaitRollout polls get_deployment with backoff, prints each node's outcome, ends only on an ending status, since a lagging node still shows the origin's earlier one, stops at once on a refusal and times out at ten minutes per node. followRollout reuses the deploy's connection and token, ends each read at the wait's deadline however slowly its reply trickles, and the CLI exits non-zero; wait/wait_timeout are kept off the wire and checked first. A read the wait's own end cut short is not reported as a failed read. The CI role grants get_deployment.

#3059, releases per node. list_releases (replication settled at once) reports each component's live id and kept releases with state, time and size, over every component with a tree or a build; replicated: true adds each node's answer. A displaced release is marked kept in its .complete content. Registered in hdbTerms, serverUtilities (import), operation_authorization (import) and the MCP schema and description.

#3061, where a deploy came from. The OIDC exchange signs the policy and run claims into the token (createOperationToken), validation lifts it onto the principal (claim), and impersonation carries it. deploymentSource picks attested over declared and forces a declared one to say so; deployComponent records it on the row. The CLI declares the Actions run and the checkout commit (wired). list_deployments filters by sha.

#1719, entry points. missingEntryPointsWarning checks each declared pattern, reading negations as exclusions, and its glob walk, with the loader's only: filters, stops at the first match. It runs in the deploy's validateCandidate, before the swap, and the response message carries the warning.

#2681, failed payloads. sweepFailedPayloads ranks every node's failed rows per project but drops only its own, applies the bounds (config names), runs after a failure is recorded and once at boot.

#3096, an upload whose pipe loses its destination. watchForAbandonment re-checks a part held after the response after its own unpipe() or when it loses a data listener, armed from the response hook.

Design notes. components/DESIGN.md gains a section on the deployment record (each item above) and one on entry points; also server/DESIGN.md, security/DESIGN.md and the index.

Docs. HarperFast/documentation#751 documents these changes for 5.4.

✅ Verification

• New unit tests fail on base and pass here, on both engines where storage is touched.
• Integration, one node: deploy-tracking checks Track per-application deployment and restart timestamps #564 and Record where a deploy came from: the commit, the CI run and the trust policy #3061; canary-rollout checks the rolling_out → success record and runs harper deploy restart=rolling to exit 0; deploy-setup-github-actions checks the role. minor-upgrade gains a check that turns on once package.json reaches 5.4.
• Three nodes, harper-pro main built on this branch: every peer takes a certified rolling release, and a peer whose canary rejects ends the record partial naming it while the others serve it. Before the activation fix, no peer did. Rerun at 5c2d664: unchanged.
• PR QA: skipped @ 11f0a2b — no dispatch QA run for this PR.

Three-node scenarios and gates

• A Linux container, with this branch's CLI. wait_timeout, wait=false, an origin killed mid-rollout (settled partial at its restart), the entry-point warning, list_releases, per-node get_components, the sha filter and failed-payload expiry all behave as documented.
• At 5c2d664: affected suites pass on dist, LMDB and type-stripped source.
• After the activation fix: the bin, components and serverHelpers unit suites pass (3,100; OptionsWatcher fails on base here too), and canary-rollout passes 11 of 11.
• Earlier gates on this Mac: test:unit:main 7707 passing; its 3 failures pass alone or fail on base too. integrationTests/deploy, components and upgrade ran in batches: only shutdown-drain-e2e failed, as on base. Type check and lint:required pass.

Unit tests by issue

• The deployment recorder's internal _recorder_finished event leaks into event_log and SSE streams #3062: recorder, tail.
• activated_from is never registered on hdb_deployment tables that already exist #2659: directive.
• Track per-application deployment and restart timestamps #564: load times and get_components, a failing search and a long unreleased run.
• A rolling deploy's record says success even when another node rejects the release #3060: peer outcomes on the row, a final write that cannot land, the lock (on a real table), boot recovery from the job.
• harper deploy restart=rolling exits 0 before the other nodes have taken the release #3058: awaitRollout, CLI, a trickling read (httpRequest).
• Report which deployment is live on each node, and which releases each node keeps #3059: real deploys, then a stage.
• Record where a deploy came from: the commit, the CI run and the trust policy #3061: exchange, shape, recorded by deploy_component, CLI.
• deploy_component: validate declared entry points survived extraction #1719: check and deploy_component, only: filters.
• Failed deploy payloads are retained forever; add age/count expiry so retried rollouts stop consuming tenant storage quota #2681: count, age, scope.
• Discard the rest of a multipart upload whose route's pipe loses its destination after the response #3096: destination error and unpipe().

Fixes #3062
Fixes #2659
Fixes #564
Fixes #3060
Fixes #3058
Fixes #3059
Fixes #3061
Fixes #2681
Fixes #3096
Refs #1719

_{🤖 Generated by Claude Code (Claude Opus 5.5); posted via @dawsontoth.}

Release target: main — new operations, fields and CLI behavior for v5.4
Related PRs: #372 overlaps (merged into this branch; its status-registry rework now carries loadedAt across a level change), #3360 overlaps (both add a 5.4.0 directive and edit operation_authorization.ts; whichever lands second merges the directives), #3498 overlaps (merged into this branch; it restarts each peer in turn for an uncertified rollout, this activates each peer for the certified one), #3339 overlaps (merged into this branch; both edit deployComponent and bin/restart.ts), #3131 overlaps (both add tests to unitTests/components/deploymentRecorder.test.js; whichever lands second merges them), #3523 overlaps (merged into this branch; both edit the root DESIGN.md index), 21 others independent
Complexity: complicated

Deferred findings

• f07ebe9f80c92088 minor data-integrity — resources/Table.ts — replica admission and demotion drop receivedIdentityRefs, weakening duplicate-replay protection — filed Replica admission and demotion drop a row's receivedIdentityRefs #3531
• f6003a95384597a3 nit reliability — components/deploymentOperations.ts — same-millisecond buffered live event dropped during replay — filed get_deployment's live tail can drop an event emitted in the same millisecond as the last replayed entry #3500
• 2ac1068bf06999ab nit reliability — components/deploymentRecorder.ts — finished guard set before the write blocks the failure path's finish('failed') — filed A failed terminal write leaves a deployment row mid-phase: finish() is already latched #3501
• 711b3ce1ee5d8f30 nit reliability — bin/restart.ts — replicated restart_service on core alone throws from the replication stand-in — filed A replicated restart_service on Harper core alone ends ERROR instead of restarting this node #3508

_{Review-Coverage: authored=claude; ran=gemini,codex,cursor-composer; adjudicated=domain; blocked=cursor-grok(no-receipt),cursor-muse(no-receipt); declined=cursor-kimi,muse; rounds=10; full=4 @ 11f0a2b}

_{Review-Attention: deep ~60m (critical: operationDescriptions.ts, operations.ts +7; sensitive: tokenExchange.ts, credentialProvenance.ts +2; decisions: rollout-row-lock-fallback, final-write-fails-job, rollout-wait-default-on, partial-status, cleanup-owner-identity, restarted-at-naming, do-less-alternative; raised: degraded review) @ 11f0a2b}

[gemini-code-assist]

Code Review

This pull request implements a rollout tracking mechanism for rolling deployments, introduces the list_releases operation to report live, staged, and kept releases on disk, and enhances get_components with deployment and worker load times. It also adds failed payload retention to expire old failed deployment tarballs and records deployment sources (attested via OIDC or declared by the client). The review feedback suggests parallelizing independent I/O operations in keptReleasesOf and releasesOnThisNode using Promise.all to improve performance. Additionally, it recommends adding defensive nullish checks and optional chaining to prevent potential TypeError crashes, and resolving TypeScript compilation errors related to handling caught errors of unknown type in deployRollout.ts.

[kriszyp]

_{🤖 Reviewed with Codex}