[kriszyp]

⊙ Problem

Redirector’s routing tests can hang on a public-registry install or on unbounded HTTP waits. The new fixture note needs a root-index entry that fits the index's byte ceiling.

🤔 Keep real installation coverage?

I dunno if I really like this approach, seems like verifying that Harper can do a real npm install is a good thing to check, even if it can possibly cause red when GH is down. I'm inclined to decline/close this PR. WDYT?

Recommendation: keep routing tests isolated and retain separate installation coverage.

💡 Solution

Bundle pinned PapaParse, cap requests, and assert that npm never starts. Add one short root-index entry. Production behavior is unchanged.

🔧 Changes

• DESIGN.md adds one index entry for the fixture note (67,209 bytes, under its 68,166-byte ceiling after Trim the root DESIGN.md index back under its byte ceiling #4036 and Keep 40% of room under a DESIGN.md ceiling, and make a forced trim cut it to 60% #4046).
• The fixture design note now covers all integration subdirectories and links its motivation instead of preserving CI-run history. The root index uses the broader heading too. Original fixture and request-timeout changes are described below with their diff links.

🤔 Timing suggestions

The skip-install log is written before deployment returns, so the helper’s fresh 15-second wait is enough. The two seed requests have no demonstrated path to the logger’s 10,000-entry throttle, so keep the check where it is. The readiness loop’s fixed 500 ms pacing is bounded: each poll times out, and the 60-second deadline throws.

🤔 Retained constraints

fetchWithTimeout stays a byte-identical copy of early-hints.test.ts's. Share it when sendOperation gets a timeout.

✅ Verification

• Rebased onto 31a0e39e5, which carries Trim the root DESIGN.md index back under its byte ceiling #4036 and Keep 40% of room under a DESIGN.md ceiling, and make a forced trim cut it to 60% #4046's root-index trims and ceiling headroom rules (DESIGN.md is 67,209 bytes, under its 68,166-byte ceiling). Locally after npm ci and a build: redirector integration (23/23), check:design-docs, check:design-docs --base origin/main, lint:required and Prettier pass.
• End-to-end route: redirector integration, 23/23 passing on the rebuilt, rebased head 17e2667.

Broad local-run limitations

Earlier broad local runs were limited: core timed out at 900 seconds; integration had 2,586 passes, six loopback-collision failures, and six Ollama JSON-import cancellations. Both collision-affected files passed separately (10/10); the shared-temp credential probe remains a local harness failure.

Related scan: partial (all open PRs, first 50 of 342 merged). The overlaps are documentation edits; none supersedes the redirector fix. #3856 merged on main, superseding the branch's earlier resources/DESIGN.md compatibility line; #3899, #4036 and #4046 merged main's design-docs fixes and root-index trims.

PR QA: skipped @ 17e2667 — test fixtures and internal notes only.

Docs companion: none; this changes test infrastructure and internal notes only.

Original implementation, alternatives, and historical verification

main went green → red at 576d333 — a release-only commit that changes three lines of package.json and package-lock.json — on one of 36 Integration Tests legs (run 35740846737, Integration Tests 6/6 (Node.js v26)). The Component: redirector suite's before hook died after 307,394 ms with UND_ERR_HEADERS_TIMEOUT, cancelling all 23 tests. I re-ran that leg on the same SHA and it passed, so main is green again; this PR removes the cause rather than the symptom.

The suite deploys template-redirector-3.0.1.tgz, whose manifest pins papaparse and which ships no node_modules, so deploy_component runs a real npm install against the public npm registry on the test's critical path. Nothing bounds that: Harper's spawn budget there is DEFAULT_COMMAND_TIMEOUT_MS = 60 * 60 * 1000, and every request the suite made was a bare fetch whose only limit was undici's implicit ~300 s headers timeout. The instance log for the failed run shows the server going silent 1.9 s after the npm child was spawned and never speaking again.

The fix bundles papaparse in the archive so deployment spawns nothing. Every request in the suite gets an enforceable deadline.

For the human reviewer

Framing-Verdict: chosen-approach-sound (planning review, codex/xhigh). No alternative was overruled or adopted; the reviewer independently derived the same layer (fixture + harness, not Harper's production deploy defaults) before reading the plan.

Decisions raised in review and resolved rather than implemented, each cheap to revisit:

• The same flake is still live in two other suites. integrationTests/mqtt/mqtt.test.ts and integrationTests/components/acl-connect.test.ts deploy fixtures pinning @harperdb/acl-connect and jsonwebtoken with no committed node_modules, through the same unbounded sendOperation. Left out of this PR to keep the red-main fix reviewable; recorded in the design note so the next timeout there is not diagnosed as a new bug.
• sendOperation itself still has no timeout, across 118 call sites in 25 suites. Adding one in @harperfast/integration-testing would fix all of them at once and would have been the wider fix; this PR only stops using it in the one suite that was failing.
• fetchWithTimeout is copied verbatim from early-hints.test.ts:21-34 rather than moved next to waitForLog.ts. Two copies can drift. I kept them byte-identical so a later consolidation is mechanical.
• The fixture is a binary archive that must be regenerated by hand. Plain npm pack template-redirector@3.0.1 would silently drop the vendored tree and restore the flake; that is why the filename diverges from the package it came from, and why the test asserts the skip-install path rather than trusting the archive.
• Nothing enforces the invariant across fixtures. A static check over integrationTests/**/fixtures that fails on production dependencies with no bundled node_modules would have caught the two suites above. Cheap to add later, out of scope here.

Declined review nits, all in fetchWithTimeout, all inherited from the early-hints.test.ts copy: the helper names a timeout only during the headers phase (a body stall rejects with a plain TimeoutError, so the test still fails, just less legibly); { ...init, signal } overwrites a caller-supplied signal, which no caller passes; and the 90 s deploy cap is below undici's implicit ~300 s, which is deliberate — redirector no longer runs an install, so it does strictly less work than early-hints under the same budget.

Changes

integrationTests/fixtures/template-redirector-3.0.1.tgz is removed and replaced by template-redirector-3.0.1-vendored.tgz: byte-for-byte the same published template-redirector@3.0.1 contents plus package/node_modules/papaparse/, which is the published papaparse@5.5.3 — the exact version the fixture's own manifest pins — with its MIT license, no symlinks, and no other package. It was packed with tar --sort=name --mtime=... --owner=0 --group=0 --numeric-owner --format=ustar | gzip -9 -n, so regenerating it reproduces the same bytes. 28,894 → 75,196 bytes. With node_modules present, installApplication takes its early return and the deploy runs no package manager at all.

integrationTests/components/redirector.test.ts stops calling sendOperation for the deploy — that was the exact call that hung — and issues it through ctx.harper.operationsAPIURL with a 90 s deadline and an explicit status check before parsing. Every other request in the suite now goes through a fetchWithTimeout helper carrying AbortSignal.timeout. That is not only defensive: the readiness poll declared a 60 s deadline but checked it after each fetch settled, so a single hung request made the deadline unreachable. The before hook now also asserts, from the instance log, that the deploy logged already has node_modules; skipping install and no [redirector:spawn:npm] line, which is what stops a future fixture refresh from quietly reintroducing the registry. That assertion reads debug-level lines, so the suite pins logging: { level: 'debug' } instead of inheriting the harness default.

integrationTests/DESIGN.md is new and DESIGN.md gains its index line: one note recording why a deployed fixture must not install from the registry, the four hermetic patterns available, which one this fixture uses, and the two suites that still violate it.

Verification

End-to-end route: the existing integration suite, extended with the skip-install log assertion.

• npm run build, then npm run test:integration -- integrationTests/components/redirector.test.ts — 23/23 pass, and suite wall time drops 26.5 s → 14.5 s, the npm install being the difference.
• Fails-on-base is established by the failing run's own log rather than a synthetic revert: at 576d333 the instance logged [main/0] [debug] [redirector:spawn:npm]: Executing npm install --force --omit=dev --no-audit --no-fund --ignore-scripts and never logged the skip line, so both new assertions fail on the old fixture.
• npm run test:integration -- "integrationTests/components/**/*.test.ts" — 149/149 pass.
• npm run test:integration -- "integrationTests/deploy/**/*.test.ts" — 85 pass, 1 skipped, 0 fail.
• npm run test:integration:all — 2159 tests, 2135 pass, 0 fail, 18 skipped. The 6 cancelled belong to integrationTests/server/ollama-backend.test.ts, which self-skips unless a local Ollama is reachable; on this machine it is, and it fails on an unrelated ERR_IMPORT_ATTRIBUTE_MISSING loading json/systemSchema.json under Node v26.2.0. Untouched by this diff and skipped in CI.
• npm run format:check, npx oxlint --deny-warnings on the changed test file, and npm run check:design-docs all clean.

PR checks only run Node 24, so the failing leg (Node 26, shard 6) is not reproduced by this PR's own checks.

Origin — the dispatch brief this PR was written from

harper main is red: Integration Tests at 576d333

main on HarperFast/harper went from green to red. Failing workflows: Integration Tests at 576d333.

Find the merge that broke it and get main green. Walk that workflow's recent push runs on main oldest-to-newest to find the FIRST red head SHA and the PR it came from, then pull the failing test names from that run and from the current head. A failure on every matrix leg (Node version, Bun, uWS, Windows) is a regression; one leg is usually a flake.

Then either fix it or, when the culprit is a single merge and the fix is not obvious, open a revert and say so — main being green is worth more than the change being preserved. Record the fingerprint on the initiatives/ci-health board doc either way (prose row beginning with the test path, never a bare ref), and add your PR to initiatives/testing-and-deploy > "CI & build infrastructure" as a bare-ref line so it renders as a card.

_{Dispatch: task main-red-kriszyp_harper_576d33330_f0efc59e · queued by automation · ran by claude/opus/high · worker kzyp-xps-1}

_{🤖 Generated by OpenAI Codex; posted via @kriszyp.}

Release target: v5.4 — preserve this PR’s current milestone; test-infrastructure maintenance.

Related PRs: #562 overlaps, #3527 overlaps, #3576 overlaps, #3856 overlaps, #3899 overlaps, #4036 overlaps, #4046 overlaps, 62 others independent
Complexity: medium

Deferred findings

• f51259a1ca0a2c49 nit reliability — integrationTests/components/redirector.test.ts — readiness loop uses fixed sleep pacing and a manual deadline instead of a condition-wait helper — NOT FILED
• 28cfd97cfbfb87d7 nit test-coverage — integrationTests/components/redirector.test.ts — Hosts seed POST response status not checked — NOT FILED
• a03f2e67169f0ea5 nit reliability — unitTests/resources/transaction.test.js — sibling out-of-order tests still base versions on Date.now() after a put with small offsets — NOT FILED

Late Grok findings — 6 surviving (highest minor) — Grok reviewed e26ad09

• minor — integrationTests/components/redirector.test.ts:116 — Startup or fixture-seeding log throttling fails redirector setup even when deployment correctly skips npm (previously adjudicated)
• minor — unitTests/resources/transaction.test.js:481 — Wall time falling more than 2 seconds behind the stored version exhausts the new wait before audit assertions run (previously adjudicated)
• nit — integrationTests/DESIGN.md:35 — Fixture guidance omits deployment-window scoping and checking logs before fixture seeding (previously adjudicated)
• nit — integrationTests/components/redirector.test.ts:69 — Optional added-comment cleanup remains at the logging setup, deploy timeout, and transaction-version setup (previously adjudicated)
• nit — resources/DESIGN.md:136 — An unrelated compatibility promise is restored without establishing which stage it describes (previously adjudicated)
• nit — integrationTests/components/redirector.test.ts:20 — The timeout helper duplicates its sibling implementation, creating two maintenance sites (previously adjudicated)

Not standing: 1 dropped-factually-wrong. Grok ran 2m (3m from launch to report); the fold adjudication (cursor-grok@e26ad09002f9-abb0a1fa, codex) took 3m.

Late Grok findings — 5 surviving (highest minor) — Grok reviewed 25eac9a

• minor — unitTests/resources/transaction.test.js:481 — the fixture waits for wall time to catch up to the record version, and a backward clock step of 2 s or more makes the wait time out before any of the patches run (previously adjudicated)
• nit — unitTests/resources/transaction.test.js:479 — the new comment states a storage invariant that the store does not keep (previously adjudicated)
• nit — resources/DESIGN.md:145 — the compatibility note ends with "no on-disk or on-wire identifiers change in this stage" (previously adjudicated)
• nit — integrationTests/components/redirector.test.ts:112 — with debug logging on, a throttle line in hdb.log fails the suite even when npm was never spawned (previously adjudicated)
• nit — integrationTests/components/redirector.test.ts:80 — the readiness loop sleeps a fixed 500 ms between polls (previously adjudicated)

Not standing: 1 merged, 1 separate-issue. Grok ran 3m (3m from launch to report); the fold adjudication (cursor-grok@25eac9aa1d6d-f9f6b710, claude) took 1m.

_{Review-Coverage: authored=gemini; ran=cursor-composer,codex,claude,cursor-grok,muse; late=cursor-grok:adjudicated; adjudicated=domain; rounds=24; full=21 @ 17e2667}

_{Review-Attention: skim ~2m (decisions: vendor-node-modules-vs-install-command, log-scrape-proof-of-no-install) @ 17e2667}

[gemini-code-assist]

Code Review

This pull request documents design guidelines for integration test fixtures, emphasizing that they must not install from the npm registry to prevent test flakiness. To enforce this, the redirector test was updated to use a vendored fixture and asserts that no npm install is spawned. Additionally, a fetchWithTimeout helper was introduced to replace standard fetch calls in the test suite. Feedback on the changes suggests improving fetchWithTimeout to combine any caller-provided abort signal with the timeout signal using AbortSignal.any().

[claude]

Reviewed; no blockers found.

[cb1kenobi]

No confirmed blocking findings were found on the changed lines. Existing discussion already covers the non-blocking timeout-signal and helper-duplication concerns.

—
Reviewed 7202e1d