Repository navigation
Run source-compatible unit tests without a build - #3199
Merged
Merged
Conversation
Run source-compatible suites without a build and retain compiled exports for legacy mutation fixtures. Preserve Mocha selection, completion and database isolation across both lanes, and cover native execution without dist in CI. Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Resolve worker helpers absolutely, classify built artifacts and cache mutation precisely, and bind result reporting to the owning run after teardown and reporter completion. Preserve ordinary parallel workers and add regression coverage for unfinished completion and physical database removal. Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request establishes a dual-lane unit testing architecture that separates native source-compatible tests (utilizing Node.js native type stripping via the typestrip condition) from legacy compiled tests. It introduces a custom test launcher (runTypeStripTests.mts) to partition suites, updates package.json imports and scripts, and implements a nativeCoreGuard to prevent source processes from loading the compiled core. Additionally, the database teardown and Mocha initialization processes are refactored to support ESM-only environments. No review comments were provided, so there is no further feedback to address.
Keep optional S3 loading compatible with native ESM, cache Bun listener binds, and handle WebSocket backpressure without a Node-only socket. Discover persisted tables when importing the public package in read-only build children. Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Give the facade probe its own configured root now that public imports discover persisted tables. Keep its real user-worker checks in the same child process and preserve completion validation. Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Keep the read-only import oracle independent of earlier resource suites and their database configuration. Verify persisted data, storage entries, file contents and completion in fresh source and compiled readers. Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Return iterators whose connect resolves after close, and settle and remove drain waiters when the transport closes. Strengthen real network coverage with stalled-reader resumption, thread identity and gated late-connect cleanup. The Bun resumption oracle exposes duplicate delivery in its runtime ws shim; transport repair follows. Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Bypass Bun ws substitution at the server boundary so paused readers resume with ordered, exactly-once frames. Keep close-settled drain waits shared by the supported socket transports. Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
This was referenced Oct 20, 2026
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
kriszyp
marked this pull request as ready for review
October 23, 2026 01:35
kriszyp
added a commit
that referenced
this pull request
Oct 24, 2026
* Add startup-phase lifecycle and runtime path helpers
Introduces utility/lifecycle.ts with onStartup(cb) / runStartup() so
side-effectful module-load work (server-singleton wiring, listener
registration, config-derived constants) can be deferred to a controlled
startup phase that runs after env.initSync() and before request handling.
This is the lever that makes ESM-cycle TDZs avoidable without restructuring
the import graph.
Adds RUNTIME_SRC_ROOT and RUNTIME_FILE_EXT to packageUtils.js. Detects
whether the running modules are .ts sources (type-strip) or .js dist files
by inspecting packageUtils.js's own __dirname, so callers like
manageThreads.startWorker() can spawn workers with the correct file path
under both modes from a single string identifier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Convert CJS .js source modules to ESM .ts
Renames the 57 .js source files that participate in Harper's runtime
module graph to .ts and rewrites them in ESM form. The driving constraint:
under Node's type-strip mode, .ts files with import/export syntax are
loaded as ESM, and any CJS .js file that require()s a .ts ESM file in the
entry's evaluation chain throws ERR_REQUIRE_CYCLE_MODULE.
Mechanical edits applied across all converted files:
- const X = require('mod') → import X from 'mod'
- const { A, B } = require('mod') → import { A, B } from 'mod'
- const X = require('mod').default → import X from 'mod'
- module.exports = { a, b, ... } → export { a, b, ... }
- module.exports = X → export default X
- 'use strict' pragma removed (ESM default)
- Relative imports given explicit .ts/.js extensions
These files keep their existing internal structure and behavior; the
conversion is syntactic, not semantic. Behavioral changes (deferred
side-effects, late-binding, etc.) live in the follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Wire type-strip entry point and defer module-load side effects
Lands the runtime support needed to run Harper directly from .ts sources
under Node's type-strip mode (node bin/harper.ts) while keeping the
existing CJS dist build (node dist/bin/harper.js) working unchanged.
Entry point (bin/harper.ts):
- Switch cases use await import('./mod.ts') instead of require('./mod')
- Splits boot into initEnv() (always cheap) and runServerStartup() (only
for start/run paths); version/help skip the heavy init
- Replaces require.main === module with a dual-mode isEntry check
Worker thread bootstrap (server/threads/threadServer.ts):
- Workers run env.initSync() → runStartup() → startServers() in an async
IIFE so they see a populated env and complete onStartup hooks before
serving requests
manageThreads.startWorker():
- Accepts an extensionless module identifier and appends RUNTIME_FILE_EXT
- Resolves against RUNTIME_SRC_ROOT (source dir under type-strip,
dist/ under CJS dist)
- listenersByType / messageListeners lazy-initialized inside the accessor
so callers don't trip TDZ when manageThreads is loaded mid-cycle
Late-binding for class-extends-Resource modules:
- ErrorResource, CertificateVerificationSource,
CertificateRevocationListSource, Login: the class is constructed
inside onStartup() and exposed via `export let X` so consumers see a
live binding after startup. These modules sit inside Resource.ts's own
static-graph SCC, so a class-extends declaration at module-top TDZ'd.
Module-load side-effect deferrals (one-off onStartup wrappers):
- server.X = … assignments in security/user.ts, security/auth.ts,
resources/analytics/write.ts, server/http.ts,
server/serverHelpers/serverUtilities.ts, server/threads/threadServer.ts
- Object.defineProperty(server, …) in server/nodeName.ts and
server/threads/manageThreads.ts
- onMessage* listener registrations in server/threads/itc.ts,
security/keys.ts, resources/analytics/write.ts,
utility/processManagement/processManagement.ts, bin/restart.ts
- Config-derived constants in security/auth.ts and
server/serverHelpers/contentTypes.ts converted to lazy getters
- server.contentTypes attachment in contentTypes.ts deferred
- whenComponentsLoaded-awaiting IIFE in
server/DurableSubscriptionsSession.ts deferred
Environment manager (utility/environment/environmentManager.ts):
- env.get() and getHdbBasePath() made defensive against being called
before this module's body has run (catches TDZ ReferenceError from
configUtils.getConfigValue and returns undefined). Belt-and-suspenders
for the rare timing window where a module-load access reaches in
before initSync completes.
- Removed 22 eager module-top env.initSync() calls; initialization is
now centralized in bin/harper.ts (initEnv) and threadServer.ts
- Top-level let/const → var so subsequent functions don't TDZ on
module-internal state if invoked through the cycle
harper_logger.ts:
- RootConfigWatcher import moved from static (which forced a
configUtils → logger cycle) to dynamic, inside the function that
actually constructs the watcher
- logRotator require replaced with dynamic import inside its setTimeout
- module.exports = {…} wrapped in `if (typeof module !== 'undefined')`
so the CJS-shaped legacy export survives tsc emit but is skipped
under ESM, where the parallel export default already covers the
default-import consumers
Compatibility fixes:
- Type-only imports tagged via tsc --verbatimModuleSyntax sweep
- CJS-only packages (lodash, fs-extra, micromatch, papaparse, etc.)
switched to default-import-and-destructure
- JSON imports tagged with `with { type: 'json' }`
- Extensionless relative imports given explicit .ts/.js extensions
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix subcommand module resolution for CJS dist mode
`(mod.default || mod)()` only worked when `await import()` returned an
ESM-shaped namespace (mod.default = function). In tsc-compiled CJS dist,
`await import('./install.js')` from a CJS file wraps the CJS exports as
mod.default, so the actual function is at mod.default.default. CI's
`Setup Harper` step (which runs the dist build) hit this with
`TypeError: (mod.default || mod) is not a function`.
Adds a small getDefaultExport() helper that walks both shapes and applies
it to the INSTALL, STOP, and STATUS cases in bin/harper.ts.
Verified locally:
- node bin/harper.ts install → reaches installer
- node dist/bin/harper.js install → reaches installer
- node bin/harper.ts version → 5.1.0
- node bin/harper.ts → "Harper successfully started."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Address claude/review feedback and unit-test regression
Issues caught by the claude/review bot and the unit-test job:
1. bin/run.ts:235 (`launch()`) — `require('../utility/processManagement/processManagement.ts')` would throw `ReferenceError` under typestrip ESM. Converted to `await import(...)` (mirrors `filterArgsAgainstRuntimeConfig` a few lines above).
2. server/threads/threadServer.ts (`onSocket`) — `require('../../components/componentLoader.ts').getComponentName` had the same problem on the secure-port code path. Converted onSocket to async and used `await import(...)`. Inspector calls in the same file (debug-mode paths) now use a single static `import * as inspector from 'node:inspector'` instead of per-call requires, so they work in both modes.
3. utility/functions/sql/alaSQLExtension.ts — `import mathjs from 'mathjs'` got `undefined` for `mathjs.mad` etc. since mathjs is ESM-only and has no default export. Switched to `import * as mathjs` and replaced the trailing `module.exports = {...}` with `export default {...}` so the file is consistently ESM. This was the cause of `TypeError: Cannot read properties of undefined (reading 'mad')` that surfaced in unit-test runs.
Verified both modes still reach "Harper successfully started."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix remaining typestrip require()s and lazy-init ErrorResource
Addresses the second round of claude/review findings plus the
ErrorResource regression Gemini flagged that showed up in unit tests:
- server/threads/socketRouter.ts:31 — `await require('./threadServer.ts').startServers()` was half-converted (extension updated but `require` left); switched to `await (await import(...)).startServers()`. Triggered when threadCount === 0 in typestrip mode.
- bin/upgrade.ts:36 — `pm2Utils = require('../utility/processManagement/processManagement.ts')` parallel to the `pmUtils` fix already done in bin/run.ts; switched to `await import(...)`. Was breaking `harper upgrade` in typestrip mode.
- utility/signalling.ts — `serverItcHandlers = serverItcHandlers || require('../server/itc/serverHandlers.ts')` in signalSchemaChange/signalUserChange; both functions made async and switched to `await import(...)`.
- server/fastifyRoutes/plugins/hdbCore.js — renamed to .ts and converted to ESM. As a CJS file it required serverHandlers.ts (now ESM), throwing ERR_REQUIRE_CYCLE_MODULE under typestrip when fastify routes load. The single importer (server/fastifyRoutes.ts) updated to point at .ts.
- resources/ErrorResource.ts — rewrote to lazy-construct the class on first
use via a Proxy. The earlier `onStartup`-based late binding worked in
production (where runStartup runs before componentLoader can construct
ErrorResource) but broke unit tests that exercise componentLoader without
going through the lifecycle, producing the `ErrorResource is not a
constructor` failure. The Proxy resolves on `new` or property access, so
the class is always available regardless of lifecycle state, and still
dodges the original module-load TDZ on Resource.
Verified locally:
- node bin/harper.ts version → 5.1.0
- node bin/harper.ts → Harper successfully started.
- node dist/bin/harper.js → Harper successfully started.
- npm run lint:required → 0 errors, 0 warnings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Preserve module-scope variable names for rewire-using unit tests
Harper's unit tests use `rewire` to override module-internal bindings
via `__set__` / `__get__`. Rewire works by `eval`-injecting code into
the module's closure, so it needs a variable with the exact name the
test references.
`tsc`'s CJS emit renames `import X from 'mod'` to a `_X_ts_1` alias
internally and rewrites all uses to `_X_ts_1.default`. After commit
1f4522a converted the underlying source files from CJS `.js` to ESM
`.ts`, the dist `_ts_1` aliases broke ~95 rewire-using unit tests
that expected to find a variable named `X` in scope.
This commit adds a `const X = _X` alias right after each affected
default/namespace/named import in the 21 source files that
rewire-using tests target. tsc emits this as a real module-scope
`const X = _X_ts_1.default;` (or equivalent), restoring the variable
name rewire looks for. ESM (typestrip) sees the same `const X = _X;`
trivially.
Files touched (each only adds the alias for the specific names the
test rewires; the rest of the imports are left alone):
bin/upgrade.ts
components/operations.ts
config/configUtils.ts
dataLayer/bulkLoad.ts
dataLayer/delete.ts
dataLayer/export.ts
dataLayer/insert.ts
dataLayer/readAuditLog.ts
dataLayer/schema.ts
dataLayer/schemaDescribe.ts
dataLayer/update.ts
dataLayer/harperBridge/lmdbBridge/lmdbMethods/{lmdbCreateSchema,lmdbCreateTable,lmdbDropTable,lmdbUpsertRecords}.ts
security/fastifyAuth.ts
server/serverHelpers/serverHandlers.ts
sqlTranslator/sql_statement_bucket.ts
validation/role_validation.ts
validation/schemaMetadataValidator.ts
validation/validationWrapper.ts
Sampled local results after applying:
- upgrade.test.js: 4 passing (was: fail in before hook)
- validationWrapper.test.js: 8 passing (was: ReferenceError validate not defined)
- insert.test.js: 16 passing
- delete.test.js: 12 passing
- sql_statement_bucket.test.js: 16 passing
- configUtils.test.js: 40 passing, 1 unrelated failure
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix unit-test regressions from CJS-to-ESM conversion
Round of follow-up fixes after the typestrip conversion exposed several
test failures in the dist (CJS-emit) build:
- utility/functions/geo.ts: turfArea, turfLength, turfCircle,
turfDistance, turfBooleanContains, turfBooleanEqual,
turfBooleanDisjoint were called as `.default(...)` (CJS interop
pattern) but the new ESM `import X from '@turf/X'` already resolves to
the default. Removed the `.default` accessors. Also switched
`@turf/helpers` to `import * as` since it only exposes named exports.
(geo unit tests now pass: 79/79)
- server/serverHelpers/contentTypes.ts: my earlier rsync was off a
pre-x-ndjson revision of main, so the rsync overwrote main's
application/x-ndjson handler. Restored it.
(contentTypes test: 16/16 passing)
- utility/signalling.ts: my prior fix made signalSchemaChange /
signalUserChange async (to `await import('serverHandlers.ts')`),
but the existing tests and callers assumed sync. Restored sync
signatures and moved the import to an `onStartup` preload + safe
optional-chain fallback for environments where startup never runs.
(signalling tests: 4/4 passing)
- utility/install/checkJWTTokensExist.ts: original CJS used
`module.exports = fn` so `require('./checkJWTTokensExist')(...)` was
callable directly. My `export default fn` converted to
`exports.default = fn`, breaking that calling convention. Added a
conditional `module.exports = fn` shim that runs only when `module`
is defined (tsc CJS emit) and is skipped under typestrip ESM.
(checkJWTTokensExist tests: 2/2 passing)
- unitTests/testUtils.js: `preTestPrep()` now calls
`lifecycle.runStartup()` after `initTestEnvironment()`. Many tests
stub server-singleton methods (e.g. `sinon.stub(server, 'getUser')`)
that are now wired during the startup phase rather than at module
load. Calling runStartup() in preTestPrep gives tests the same
fully-wired state production sees. (`runStartup` is idempotent.)
(auth.test.js: 10/10 passing)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Alias remaining imports for rewire-using tests (PropertiesReader, installer)
The rewire-target extractor missed two patterns in the first pass:
- Variable-based rewire calls like `rewire(MODULE_PATH)` where MODULE_PATH is
a local string constant
- The harper_logger PropertiesReader import (via the requireUncached helper)
Applies the same const-alias-after-import pattern to:
- utility/logging/harper_logger.ts: PropertiesReader
- utility/install/installer.ts: PropertiesReader, installValidator, mountHdb,
checkJwtTokens
Local check: harper_logger.test.js now 24/24 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Lazy-access harperBridge in dataLayer; fix __dirname use in jobRunner
Two issues uncovered while investigating integration-test regressions:
- dataLayer/{insert,delete,readAuditLog}.ts imported `harperBridge` via
`import _harperBridge ...; const harperBridge = _harperBridge;`. Under
the existing intentional cycle (insert.ts ↔ harperBridge.ts via the
bridge's transitive deps), `_harperBridge` is in TDZ at the moment
the alias evaluates, so `harperBridge` was bound to `undefined`
forever and the daemon crashed at startup with
`Cannot access '_harperBridge' before initialization`. Replaced the
capture with a Proxy that defers `.X` reads to call time — the cycle
is fully resolved by the time any usage fires, and rewire tests can
still patch the binding name.
- server/jobs/jobRunner.ts used `join(__dirname, './jobProcess.js')` to
resolve the job-worker entry path. `__dirname` is undefined under
type-strip ESM, so the call threw silently when a job was launched
(which silently meant CSV/bulk-load jobs never spawned a worker).
Switched to an extensionless identifier `'server/jobs/jobProcess'`
so `manageThreads.startWorker()` resolves it against `RUNTIME_SRC_ROOT`
with the correct extension per execution mode.
Note: integration-test CSV bulk-load is still broken locally — the job
worker startup hangs without reaching the actual data path. The above
fixes are necessary preconditions but not sufficient; ongoing
investigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Convert remaining top-level require() calls in LMDB/launch files to imports
claude/review flagged that the LMDB bridge files still had top-level
\`require('./path').default || require('./path')\` shims that throw
\`ReferenceError: require is not defined\` under typestrip ESM. Same in
the launchHarperDB entry script and mount_hdb's createTables.
Converted via regex sweep — pattern was uniform across these files:
\`const X = require('./path').default || require('./path');\`
becomes \`import X from './path';\`. Where the require pattern was
spread across multiple lines (triple-require defensive shims in
lmdbCreateTransactionsAuditEnvironment), collapsed to a single import.
launchServiceScripts/launchHarperDB.ts: top-level \`require(...)\` call
replaced with a dynamic \`import(...).then(...)\`.
utility/mount_hdb.ts: inline require inside \`createTables\` hoisted to
a top-of-file import (CreateTableObject).
Verified \`node bin/harper.ts version\` still works. \`npm run lint:required\`
clean, \`npm run format:write\` no changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop unused JSDoc type require() in lmdbSearchByConditions
The destructured SearchByConditionsObject/SearchCondition were only
referenced from JSDoc @param annotations — under typestrip mode the
top-level require() throws ReferenceError. The TS file is already
typed; the JSDoc-only imports add no value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Expose alaSQLExtension members as named exports
The pre-conversion CJS module set members directly on module.exports,
which let both alasqlFunctionImporter (alasqlExtension.distinct_array
etc.) and the unit test (alasql_extension.searchJSON) read them from
the top level. The ESM conversion collapsed everything under
export default {...}, so both code paths started reading undefined.
Promote each member to a named export and keep the default as a
convenience aggregate so both shapes work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Convert remaining ESM-file require() calls flagged by claude/review
Three stale require() calls left over from the .js→.ts conversion sweep
were still active in files that now evaluate as ESM, so any code path
that hit them threw ReferenceError: require is not defined under
typestrip mode (node bin/harper.ts):
- security/keys.ts updateConfigCert() (harper renewcerts path)
- config/harperConfigEnvVars.ts getLogger() (any HARPER_*_CONFIG env-var path)
- resources/Resources.ts Resources.set() conflicting-path fallback
In all three cases the require was a lazy load (cycle-avoidance / cold
path). Hoisting to a static top-level import is safe because none of
the targets create a cycle back to the importer, and we don't need to
defer the load — the modules are already loaded elsewhere by startup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drain startup hooks before HTTP-test server start
The unit-test apitests reach server-singleton consumers
(server.http, server.getUser, server.operation, etc.) without going
through bin/harper.ts, which is where production calls runStartup().
Without that drain, this PR's onStartup wiring stays deferred and the
test setup hits "scope.server.http is not a function" once REST tries
to register routes. Call runStartup() in setupTestApp before
startHTTPThreads so the same hooks fire in the test path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Stop subcommand env init and config-load resilience
Two related fixes for paths that bypass bin/harper.ts's normal start
flow:
- bin/harper.ts STOP case now calls initEnv() before importing
stop.ts, so systemInformation.getHDBProcessInfo can resolve the
ROOTPATH-derived pid-file path (was hitting TypeError: path argument
must be of type string).
- components/componentLoader.ts now defaults config to DEFAULT_CONFIG
when getConfigObj() returns undefined (config not initialised). This
is the same fallback the existsSync=false branch already uses and
avoids the `Cannot read properties of undefined (reading
'extensionModule')` crash in api-test setup where /tmp/hdb/harper-config.yaml
exists but initConfig has not run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Wait for whenComponentsLoaded before api-test requests
startServers() returns synchronously after firing loadRootComponents().
.then(listenOnPorts) without awaiting, so startHTTPThreads(0) resolves
before the HTTP port is bound. The api-test setup then races into
axios.put() and gets ECONNREFUSED. Await the shared whenComponentsLoaded
promise so the server is actually listening when the first request goes
out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Remove whenComponentsLoaded await in api-test setup
Awaiting whenComponentsLoaded after the threadServer fix surfaced a
hang in loadRootComponents during the api-test path that does not
reproduce on the production startup path. Reverting just the await so
the suite races the listen as before; the underlying loaded-promise
chain in threadServer is still correct for production callers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restore initSync() at top of databases.ts
main-branch databases.js called env.initSync() at module load — the
conversion to .ts dropped that call. Without it, code paths that reach
env.get(...) through configUtils.getConfigValue without going through
preTestPrep (notably the unit-test apitests, which import databases.ts
transitively but never call initTestEnvironment) leave flatConfigObj
undefined. installApplications then sees null componentsRoot and the
loader-side fallbacks could not paper over every consumer.
Restore the call so the boot-props / autodetect path runs whenever
databases.ts is loaded — same behaviour as main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drain startup before server.getUser override in api-test setup
setupTestApp captured server.getUser, then overrode it with the
test/test fake-user implementation, then called runStartup() — which
fires user.ts's onStartup hook and reassigns server.getUser back to
the real getUserImpl. The override never survived to handle the auth
header on the test's first request and the restricted-user tests got
401s instead of the seeded permission set.
Drain runStartup() before the override so user.ts's hook lands first,
then capture/replace getUser. Single drain point.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Make onSocket sync to preserve mTLS test compatibility
The .js->.ts conversion replaced require('../../components/
componentLoader') inside onSocket with `await import(...)` and marked
onSocket async. server.socket(listener, options) therefore returned
a Promise instead of the underlying TLS/net server, breaking callers
that expected a sync server handle (notably the api-test mTLS test:
`startMQTT(...)[0].listen(8884, resolve)`).
Hoist getComponentName to a static import so onSocket can stay sync
and return the actual server.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restore initSync() module-load calls (cycle-tolerant)
The .js->.ts conversion dropped envManager.initSync() top-level calls
from 17 source modules. On main those calls were what got the harper
config loaded for any code path that touched env.get() without going
through bin/harper.ts (notably unit-test files that import from
resources/Table.ts, resources/databases.ts, security/auth.ts, etc.).
The previous wrap-up posted with the apitests fixed addressed the most
visible regression (componentsRoot in api-test setup, fixed by
initSync in databases.ts). The remaining test:unit:resources segfault
traced back to the same kind of state: caching.test.js hitting an
unhandled rejection on primaryStore.cache, which left native storage
in a bad state for the next describe block.
This commit:
- Restores initSync() top-level calls in all 17 files where it was
dropped, matching main.
- Wraps each call in try/catch so typestrip ESM evaluation cycles
don't blow up at module load.
- Updates initSync() itself to silently return on ReferenceError
(cycle TDZ) instead of process.exit(1), since the same module-load
chain (or bin/harper.ts) will retry once evaluation completes.
Verified `node bin/harper.ts version` still works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Graceful worker shutdown in create.test.js to avoid rocksdb-js segfault
Worker.terminate() leaves the rocksdb-js native handles in a state
where their finalizer (during V8 GC of the worker's isolate) crashes
the next test's rocksdb open in the parent process — a
@harperfast/rocksdb-js native lifecycle bug. The bug exists
independently of this PR, but typestrip-induced timing changes
make CI hit it more reliably.
The worker already handles `{ type: 'shutdown' }` with process.exit(0),
which closes the rocksdb handles on the worker's own event loop.
Send that message and await `exit` before calling terminate as a
fallback. Local repro of test:unit:resources now reaches 385 passing
with no segfault.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Lazy dataLoader logger + lazy componentLoader in operations.ts
dataLoader.test.js stubs harperLogger.forComponent immediately before
requiring resources/dataLoader, but the stub gets defeated when an
earlier test file in the same mocha process transitively imports
dataLoader.ts — that captures the real forComponent into
dataLoaderLogger at module load and the test sees zero stub calls.
Two fixes:
- components/operations.ts: revert the top-level componentLoader
imports back to lazy `await import('./componentLoader.ts')` inside
the function bodies that actually use them. This matches the lazy
pattern main's operations.js used ("Lazy-load to avoid circular
dependency with componentLoader") and removes one path that pulls
dataLoader.ts into the test-time graph.
- resources/dataLoader.ts: defer the `harperLogger.forComponent(...)`
call behind a Proxy so the actual logger is resolved on first use
rather than at module evaluation. Other consumers (http.ts,
threadServer.ts) still pull componentLoader -> dataLoader.ts
transitively, so this is the more robust guarantee.
Local test:unit:resources now reports 386 passing, 0 failing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Move startup-hook drain into run.ts main() after initialize()
bin/harper.ts was draining onStartup hooks BEFORE run.ts's launch() /
main() finished install/initialize. auth.ts's onStartup hook calls
`table({...})` which resolves through `getHdbBasePath()` — but
hdbBasePath is only populated by the install path inside initialize().
Integration tests that spawn a fresh Harper saw the auth hook fire
against empty installProps and crash with `path must be string`
during `database()`.
Move the runStartup() drain into bin/run.ts's main() after
initialize() completes, and drop the early calls from
bin/harper.ts START / undefined-service paths. Smoke (node bin/harper.ts version)
remains green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Lazy-Proxy CertificateVerificationSource for stub-friendly construction
claude/review flagged that onStartup(() => CertificateVerificationSource =
class extends Resource { ... }) leaves the exported binding undefined in
test paths where startup has already drained before the test imports the
class. Use the same lazy-Proxy pattern as resources/ErrorResource.ts so
the export is always a valid Proxy that constructs the real class on
first use, independent of lifecycle state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Resolve apitests/CRL conflict on startServers awaiting
The previous round split between two callers:
- unit:apitests setupTestApp needs to wait for HTTP/HTTPS ports to bind
(so axios doesn't race the listen).
- Integration test workers spawned via `new Worker(threadServer.ts)`
must NOT block their bootstrap IIFE on the same chain — that
deadlocked Harper startup in CRL/integration scenarios.
Resolve:
- Restore the `return Promise.resolve(listening).then(...)` chain inside
startServers so the loadedPromise (and whenComponentsLoaded with it)
only resolves after listenOnPorts + CHILD_STARTED.
- Drop the worker IIFE's `await startServers()` — startServers
schedules its chain internally and notifies main via CHILD_STARTED
postMessage; awaiting it deadlocks.
- Have setupTestApp.mjs explicitly `await whenComponentsLoaded` after
startHTTPThreads so apitests get the listen guarantee without
relying on startServers's return value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Lazy-Proxy CertificateRevocationListSource (matches CertificateVerificationSource)
claude/review's blocker called out CertificateVerificationSource AND
CertificateRevocationListSource. Only the first one got the Proxy
refactor last round; this one was still wrapped in `onStartup(() =>
CertificateRevocationListSource = class extends Resource {})`. Apply
the same lazy-Proxy pattern so the export is always a valid Proxy
regardless of lifecycle state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Skip DEFAULT_CONFIG fallback for non-root components with empty config
CRL/OCSP integration tests' fixture/config.yaml contains only a YAML
comment, so parseDocument(...).toJSON() returns undefined. The earlier
`config ??= DEFAULT_CONFIG;` fallback (intended for root configs when
getConfigObj() returns undefined) then synthesized DEFAULT_CONFIG
(rest, graphqlSchema, etc.) for the fixture. The component loader
created a Scope for each synthesized plugin, whose OptionsWatcher
watched the fixture's empty config.yaml; with no matching plugin key
in the file, OptionsWatcher never emits 'ready', scope.ready hangs
forever, sequentiallyHandleApplication never starts, and the http
worker never sends CHILD_STARTED — so Harper times out at startup.
Constrain the fallback to isRoot (which is the case the comment
described) and treat an empty/null parse result on a non-root
component as "nothing to load" rather than silently filling in
default plugins.
Fixes the CRL Certificate Verification, OCSP Certificate Verification,
and related mTLS-bearing integration test hangs on PR #562.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix all 187 TypeScript build errors (clean tsc emit)
Key patterns fixed across 46 files:
- Dynamic properties on Worker/MessagePort/MessageChannel typed as `any`
(unexpectedRestarts, wasShutdown, startCopy, name, threadId, etc.)
- `Set<string>` in Tables index signature causes TS2339 on table lookups;
added `as any` casts at access sites
- `new Promise<void>((resolve) => resolve())` → added `<void>` generic in
5 places (manageThreads.ts, threadServer.ts)
- Missing field declarations in LMDB transaction subclasses
- Bun.serve options object widened to `any` (no @types/bun dependency)
- `node:vm` experimental API (SourceTextModule/SyntheticModule) widened
to `any` — existing @ts-expect-error was made effective
- Legacy globals (HDB_TABLE_INFO, hdbTableAttributes, hdbAttributeAttributes)
in lmdb create methods: restored eslint-disable-next-line ordering so
no-undef suppression is adjacent to the actual reference
- Removed stale @ts-expect-error directives that caused TS2578 "unused"
- Import attribute `with { type: 'json' }` not supported in CJS target;
removed from globalSchema.ts and mount_hdb.ts
- Optional parameter additions for several functions with narrow signatures
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix @ts-expect-error + eslint-disable ordering for legacy globals
The previous commit removed @ts-expect-error from HDB_TABLE_INFO /
hdbTableAttributes / hdbAttributeAttributes references, leaving TS2304
errors that break the unit-test CI (which runs tsc directly without
|| true).
Restore @ts-expect-error on the line immediately before each reference,
and move the eslint no-undef suppression to an inline eslint-disable-line
comment on the same line as the identifier. This order satisfies both:
• TypeScript: @ts-expect-error applies to the next code line
• ESLint/oxlint: inline eslint-disable-line applies to the current line
Also remove the /* global ... */ declarations that were added but don't
actually suppress TypeScript errors.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Restore rewire-compat aliases for itc.ts and serverHandlers.ts
Two unit tests failed because TypeScript compiled named imports into
prefixed CJS bindings (manageThreads_ts_1.broadcastWithAcknowledgement,
os_1), removing the module-level symbols that rewire's __set__ injects
by name.
Restore the `import X as _X; let X = _X` alias pattern for:
- broadcastWithAcknowledgement in server/threads/itc.ts
- os in server/serverHelpers/serverHandlers.ts
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix prettier formatting in databases.ts and test JSON file
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Remove accidentally committed test artifact file
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix JSON import attribute error in Node 24 typestrip mode
Static JSON imports (`import x from './file.json'`) require
`with { type: 'json' }` in Node 24 ESM/typestrip mode, but TypeScript
with module:NodeNext + CJS output raises TS2856 if the attribute is
present.
Replace all six static systemSchema JSON imports with synchronous
`JSON.parse(readFileSync(...))` using `PACKAGE_ROOT` from
`utility/packageUtils.js` for path resolution. This pattern:
- Works in typestrip mode (ESM, Node 24)
- Works in CJS dist (Node 20+)
- Emits no TS2856 errors
- Is consistent with how other JSON files are loaded in the codebase
Files changed: utility/globalSchema.ts, utility/mount_hdb.ts,
security/user.ts, dataLayer/.../lmdbSearch.ts,
dataLayer/.../lmdbDropAttribute.ts, upgrade/directives/5-2-0.ts
Closes reviewer feedback from cb1kenobi.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix mount_hdb rewire test after JSON import refactor
The unit test was injecting `systemSchema_json_1` (the CJS name tsc
emits for `import systemSchema from './file.json'`). The previous commit
replaced the static JSON import with `const systemSchema = JSON.parse(…)`,
so the rewire target is now `systemSchema` directly. Update the test to
match and drop the `{ default: … }` wrapper that the import helper added.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix missed process.exit(1) → realExit(1) in worker startup error path
The review caught that the fatal worker startup catch block still called
process.exit(1) instead of realExit(1). Since workerProcessGuard.ts is
imported at the top of threadServer.ts, process.exit is already replaced
by the guard's logging stub by the time this catch fires — the worker
would log the error but continue running in a broken state.
Every other fatal exit in this file was already updated in the merge
commit; this one was missed.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix .js import extensions → .ts for typestrip mode
Static and dynamic imports of TypeScript source files were still using
.js extensions in several files, causing ERR_MODULE_NOT_FOUND when
running under Node 24 type-strip mode (node bin/harper.ts).
Files fixed:
- upgrade/directives/5-2-0.ts: initializePaths.js → .ts
- components/deployLifecycle.ts: manageThreads.js → .ts
- components/mcp/index.ts: configUtils.js → .ts
- server/operationsServer.ts: configUtils.js, itc.js, manageThreads.js → .ts
- security/certificateVerification/certificateVerificationSource.ts:
crlVerification.js, ocspVerification.js → .ts (dynamic imports)
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Restore accidentally-deleted security/certificateVerification files
The previous commit ran `git add` without GIT_WORK_TREE set, causing
git to resolve the path relative to the git-dir root instead of the
worktree root. Only certificateVerificationSource.ts was tracked
correctly (the sed-i broke its hard-link, giving it a new inode that
git could resolve). The other 8 files in the directory appeared deleted
in the index because git couldn't open the directory.
Re-add all files with GIT_WORK_TREE explicitly set to restore the
directory to the committed state.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix regression in server/operationsServer.ts and related files
The previous .js→.ts commit used sed -i which breaks hard links by
creating a new inode. When git add ran without GIT_WORK_TREE set, git
resolved the wrong worktree root and staged an older version of
operationsServer.ts (from the main checkout's hard-linked copy),
accidentally reverting:
- env.initSync() try/catch wrapper
- `type` keyword on fastify imports
- requestTimePlugin.ts and serverHandlers.ts extensions
- operationsServer default parameter `= {}`
The correct on-disk versions of all 4 affected files are now staged
using GIT_WORK_TREE=$(pwd) to ensure the worktree root resolves
correctly in this submodule worktree setup.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Replace require() calls in common_utils.ts with ESM import
In typestrip mode (node bin/harper.ts), `require` is undefined in ESM
modules. common_utils.ts had 6 lazy `require('../resources/databases')`
calls inside function bodies to break circular-import cycles.
Replace with a top-level ESM import of getDatabases and a thin wrapper
function. The circular dependency is safe because these functions are
only called at runtime, never during module initialization.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix circular import breaking restart listener in CJS dist
common_utils.ts → databases.ts → itc.ts → common_utils.ts created
a circular dependency that caused the 'restart' ITC listener to not
register on the main thread, breaking restart_service jobs (workers
couldn't restart HTTP workers after component hot-reload).
Fix: inline isEmpty() directly in itc.ts (it was the only function
itc.ts used from common_utils) so itc.ts no longer imports common_utils.
This breaks the cycle and allows common_utils.ts to safely add a
top-level import of getDatabases from databases.ts.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix require-in-ESM via databasesRef registry instead of circular top-level import
Previous approach (direct top-level import) created circular deps with
multiple modules: databases.ts → Table.ts / auditStore.ts / DatabaseTransaction.ts
→ common_utils.ts, all of which import back into common_utils.ts.
New approach:
- utility/databasesRef.ts: tiny dep-free registry with setDatabasesGetter()
and getDatabases()
- resources/databases.ts: calls setDatabasesGetter(() => getDatabases()) at
module top-level, after its own getDatabases export is defined
- utility/common_utils.ts: imports getDatabases from databasesRef.ts (no cycle)
- server/threads/itc.ts: inlines isEmpty to remove its import of common_utils
(that import would have created a cycle via databases→itc→common_utils)
This eliminates all `require()` calls from common_utils.ts while keeping the
lazy-loading semantics needed to avoid circular dependency issues.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix validate.js default-export unwrapping in ESM typestrip mode
In ESM (node bin/harper.ts), `import _validate from 'validate.js'`
returns the CJS module wrapped as `{ default: validateFn }`. The old
`import * as validate from 'validate.js'` therefore left `validate`
pointing at the namespace object, making `validate.cleanAttributes`
undefined.
Use `_validate.default ?? _validate` to handle both ESM (where .default
is the library) and CJS dist (where _validate already IS the library).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Fix typestrip drift + open review blockers
- resources/RecordEncoder.ts: stale environmentManager.js import -> .ts (main
added this import after the conversion; ESM needs the real extension). This was
the only stale .js->.ts import; verified via full-tree scan.
- unitTests/dataLayer/bulkLoad.test.js: rewire binding _js_1 -> _ts_1 (tsc CJS
emit names the import after the .ts source file) [open blocker 1]
- unitTests/resources/caching.test.js: cache.clear() -> cache?.clear() to match
Table.ts's optional-chained cache access on the RocksDB path [open blocker 2]
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix ESM require() calls and componentLoader duplicate block
- bin/cliOperations.ts: replace require('dotenv') with static import
- bin/harper.ts: replace require('./mcp') with await import('./mcp/index.ts')
- bin/restart.ts: replace both require('./run') with await import('./run.ts')
- components/componentLoader.ts: remove duplicate null-config guard block (merge artifact)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix harper_logger.ts ESM export and segfault handler; remove lock file
- export default was missing start/startOnMainThread (updateLogSettings),
so YAML logging config was silently ignored in TypeStrip mode
- replace require('segfault-handler') with dynamic import() so it works
in ESM without require being defined as a global
- remove .claude/scheduled_tasks.lock from git tracking (tooling artifact)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix harper_logger.ts ESM export missing start/startOnMainThread; fix segfault handler
In TypeStrip (ESM) mode, componentLoader calls extensionModule.startOnMainThread?.()
on the harperLogger plugin to trigger log settings from YAML. The export default
object was missing start and startOnMainThread, so log configuration was silently
ignored in TypeStrip mode. Add both pointing to updateLogSettings.
Also replace require('segfault-handler') with dynamic import() so it works in ESM
where require is not available as a global (the try/catch suppressed the crash but
left the segfault handler unregistered).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix worker SHUTDOWN handler updating wrong restartNumber binding
The CJS→ESM conversion turned module.exports.restartNumber = ... into a
globalThis assignment instead of reassigning the exported module binding,
so restart-interrupt checks in databases.ts and threadServer.ts (which
read the real live binding) never observed a pending restart. A
long-running index backfill would run to the 10s force-kill instead of
checkpointing gracefully.
Also confirms the threadServer.ts realExit(1) fix (flagged in the same
PR review round) is already in place on this branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Preserve current runtime behavior in TypeStrip mode
Retain upstream startup, strict ITC acknowledgments, storage behavior and subprocess policy while completing the source ESM migration. Preserve compiled dispatch interfaces and add native/compiled runtime and REST worker regression coverage.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Clarify TypeStrip startup and dispatch comments
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Skip nonexistent eval entries during worker startup
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve compiled runtime paths through symlinks
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep worker fixtures out of server test discovery
Cover startup ordering, idempotency, reset, late registration and failure in both execution modes.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Refresh cached configuration before runtime startup
Re-arm transaction monitoring and token lifetimes after the ESM graph links. Run configured startup in job workers and exercise worker restrictions and transaction monitoring in both runtime modes.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep CLI imports safe in evaluated workers
Check the entry exists before resolving its canonical path, and import both CLI and server modules in the compiled and TypeStrip worker regression.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve cold SQL loading and startup logging in both runtimes
Keep the accepted worker heap minimum, synchronous SQL authorization and lazy legacy Fastify loading. Restore cold SQL module boundaries with symlink-safe runtime resolution, refresh the operation logger after installation, and correct native CSV parser interop. Exercise heap startup, logging, SQL exports and authorization through real compiled and native processes.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve TypeStrip startup and public runtime boundaries
Keep nested initialization inside the startup barrier, expose the declared
source facade values, and carry source conditions into managed workers.
Restore component errors, configured analytics and logger sink behavior;
verify cold SQL table and attribute authorization in both runtimes.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve main startup behavior and CSV source loading
Restore main's empty-component skip and hold HTTP workers alive across
startup hooks. Preserve both legacy JavaScript launcher paths through the
TypeScript implementation. Use PapaParse's actual default in the shared
CSV stream helper.
Verify both modes with real workers, launcher authentication and CSV rows.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve configured codecs and terminal startup behavior
Select direct JSON codecs before serving requests, restore serial log rotation reconfiguration, and drain interrupted HTTP startup before exit. Verify real cache reads and dual-mode REST/subscription behavior.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep response compression and shutdown continuations direct
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* fix: initialize fresh-install configuration at startup
Refresh newly eager authentication, application-loader, MQTT, forwarding,
bulk-load and debugger state after in-process installation. Keep the
existing configuration expressions and initialize the inspector once.
Exercise the real fresh-install main thread and actual cache timer in
compiled and TypeStrip modes. Resolve the security fixture after startup.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Fix main-thread inspector configuration overrides
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep framing coverage for the renamed backup loader
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve TypeStrip startup across current main changes
Retain main's lazy dependencies, canary admission and dedicated worker pools. Keep loading/reporting separate from readiness so refused or stopped workers finish disposal, and cover both runtime paths with real worker fixtures and a source-import guard.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Normalize formatting after the TypeStrip rebase
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Fix embedded core runtime paths and schema emission
Preserve product startup and public relative-worker paths while locating core workers and shared schema assets from their own module tree.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Resolve lazy core plugins independently of product imports
Preserve the caller module family under symlinks and restore extensionless compiled CLI entry detection. Exercise default embedded plugins, the legacy launcher and both CLI spellings.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Run source-compatible unit tests without a build (#3199)
* Split source and compiled unit test runtimes
Run source-compatible suites without a build and retain compiled exports for legacy mutation fixtures. Preserve Mocha selection, completion and database isolation across both lanes, and cover native execution without dist in CI.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Use portable separators in compiled runtime fixture
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve fixture selection and completed Mocha runs
Resolve worker helpers absolutely, classify built artifacts and cache mutation precisely, and bind result reporting to the owning run after teardown and reporter completion. Preserve ordinary parallel workers and add regression coverage for unfinished completion and physical database removal.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Complete empty workers and stop interrupted Mocha children
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep native CI action pins aligned with their versions
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Verify completion inside actual parallel workers
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Share source package bindings for blob encoding fixtures
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Use the named schema handler in native branch fixtures
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Drop unused lane selection metadata
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: avoid native loader hooks on supported Node floors
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: keep teardown regression in the native lane
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: preserve initial roots and complete native teardown reliably
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* docs: name the supported Mocha installation
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: resolve source types and clean parallel worker roots
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: keep harness types scoped to the launcher helpers
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: drain loader warnings before the log fallback oracle
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: keep skipped schema fixtures from changing the process PID
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: preserve completion metadata and storage shutdown ordering
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* docs: clarify unit runtime selection and configured timeouts
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: pass an absolute backup repository path to restore children
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* fix: restore Bun and embedded build integration startup
Keep optional S3 loading compatible with native ESM, cache Bun listener binds, and handle WebSocket backpressure without a Node-only socket. Discover persisted tables when importing the public package in read-only build children.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: isolate public facade and worker import probes
Give the facade probe its own configured root now that public imports discover persisted tables. Keep its real user-worker checks in the same child process and preserve completion validation.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* test: persist package import fixtures in a private process
Keep the read-only import oracle independent of earlier resource suites and their database configuration. Verify persisted data, storage entries, file contents and completion in fresh source and compiled readers.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* fix: clean up WebSocket subscriptions after disconnect
Return iterators whose connect resolves after close, and settle and remove drain waiters when the transport closes. Strengthen real network coverage with stalled-reader resumption, thread identity and gated late-connect cleanup. The Bun resumption oracle exposes duplicate delivery in its runtime ws shim; transport repair follows.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* fix: use installed WebSocket transport for Bun backpressure
Bypass Bun ws substitution at the server boundary so paused readers resume with ordered, exactly-once frames. Keep close-settled drain waits shared by the supported socket transports.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* fix: defer uWS drain notifications until native callbacks return
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
* perf: reuse the uWS drain callback per connection
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
---------
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
* Adapt incoming transport and Fastify fixtures to TypeStrip
Use a real Node socket for the existing MessagePack subscription fixture's
listener cleanup, and unwrap the Fastify plugin's default export in both
module modes. Keep the plugin probe eligible for the native source lane.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Defer model persistence until core initialization completes
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep mutable logger export fixture in the compiled lane
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Preserve protected core names through cold startup
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Fix main's red eviction test: build its version-reusing write from an absent-key merge (#3471)
* Build the eviction resequence test from an absent-key merge, not a stale overwrite
The #3384 regression test created its version-reusing write with an older patch to a key
the record already held. That only merged because of the audit:false bug #3411 fixed
(a stale patch whose keys all existed re-applied in full), so main went red once both
landed. An older patch that adds a key the record lacks still merges under the read
version, and the caller's copy then lacks the merged index key entirely.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-main-red-evictionblobs-3411-20261023
* Name the merge rule the resequence test's seed depends on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: harper-main-red-evictionblobs-3411-20261023
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 582b12dac6a799f1c2950508378dde90ac455c81)
* Make TypeStrip runtime fixtures portable to Windows
Use file URLs for trusted fixture plugins, the native path separator for distribution checks, and normalized stack paths for config-reader attribution. Preserve the existing admission and shutdown assertions.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Guard source imports and merged login body handling
Reject runtime imports of deleted JavaScript files with TypeScript siblings, while preserving erased type imports. Cover multipart credential collection and nullish bodies through the lazy login resource.
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Isolate certificate reload fixtures and await settled MQTT wills
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep mutable worker-pool logger tests in the compiled lane
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Keep mutable residency logger tests in the compiled lane
Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
* Pin admission fixtures to an indexed IPv4 listener
Use the same explicit IPv4 listener for reservation, admission and bind-retry probes, and supply the worker index required by exclusive ownership. Capture worker stdout so its startup banner does not alter the parent proof marker.
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
* Allocate available loopback ports for uWS test listeners
Replace PID-derived ports and adjacent offsets with kernel-selected free IPv4 ports for every TCP fixture. Keep the same payload, peer, backpressure and disconnect assertions.
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
* Partition compiled core CI into bounded directory suites
Run core, component and server files in complementary Mocha selections while retaining the canonical local main command and every existing step limit. Keep the job backstop above the corrected maximum sum of bounded steps.
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
* Import the Mocha constructor portably in the shutdown fixture
Node 22 exposes a namespace from require(mocha), while Node 26 returns its constructor. Use the public ESM default before loading the unit harness, keeping the root-lifetime assertions and timeout intact.
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
* Recheck shutdown after the failed-closed scan at startup
Dispatch-Task: rebase-merge-kriszyp-harper-562-fbcb11d4bd93
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Load fixture workers' runtime modules through #src so the source lane can run them
Dispatch-Task: rebase-merge-kriszyp-harper-562-fbcb11d4bd93
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: GPT-6.1 Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⊙ Problem
Run ordinary tests from source; keep export rewriting compiled. Fix Bun startup/backpressure and Next.js table discovery.
Stacked on #562; merge that foundation first.
🤔 Retain legacy mutation fixtures?
Keep legacy mutation coverage compiled;
#jsdeclares it.💡 Solution
package.json selects source through
#srcundertypestrip, denying#js. Ordinary Mocha usesdist; mixed runs use separate processes.⚖️ Alternatives
Automatic helper selection favors the launcher over fixed lists.
🔧 Changes
Product and architecture tour
Choose the runtime
npm run test:unit:typestripnpm run test:unit:typestrip:resourcesnpm run test:unit:typestrip:maintypeStripSuites.mts traces
#jsthrough helpers. runTypeStripTests.mts preserves Mocha options and compiled global setup, rejecting mixed native conditions. Use project Mocha; run external/file reporters per lane.Published exports stay compiled.
Completion and storage
mocha.init.js checks completion, totals and signals.
--fail-zerochecks the combined total.perFileDatabaseTeardown.js cleans reopened ESM databases, preserving initial directories. nativeModules.mts rejects compiled cache entries; utility-only suites leave storage unloaded. packageBindings.mts preserves blob/addon identity. rocksdbBindings.mts delegates to it. testUtils.js initializes native stores;
.mtsavoids rewire’s hook.Bun and embedded builds
AWSConnector loads the optional SDK lazily from ESM. awsSdkLoader uses the existing native package resolver; bulkLoad and export follow the rename. Dependency notes cover installation.
threadServer shares Bun's bind promise. HTTP transport selects installed ws on Bun via loadNativePackage; Bun's substitute duplicated frames after a paused reader resumed. REST settles drain waits on close and returns late subscriptions. Loading notes record resolution. The uWS adapter defers drain until its native callback returns and buffer empties, preventing a resumed sender from stalling.
The public import discovers tables without startup. Set root/read-only before import. Startup notes record these boundaries.
Decisions
🤔 Use the installed ws server on Bun?
Yes. Bun 1.4.2 duplicated queued frames. The shipped ws library preserves socket/drain semantics, at JavaScript framing cost.
🤔 Discover tables when the public package is imported?
Yes. Next.js builds need tables when import resolves. Synchronous discovery restores that contract. Configure root/read-only before import.
🤔 Defer uWS drain?
Yes. A buffer-empty check alone still stalled. One coalesced Immediate exits the native callback; Node and Bun retain their timing.
Additional changes
alasqlFunctionImporter.ts fixes native CJS date functions. DESIGN.md records that boundary. unit-test.yml adds no-build/no-install source CI and harness checks. tsconfig.harness.json scopes that check.
AGENTS.md maps both lanes; CONTRIBUTING.md documents limits; DESIGN.md records the lane contract; DESIGN.md indexes both notes. hnsw-real-store.cjs keeps its source family.
✅ Verification
S3 probes construct clients. Read-only imports verify data, unchanged storage and idle startup in both modes/engines. Facade probes isolate worker roots. Runtime integration and its pressure fixture exercise live subscriptions, disconnect cleanup and actual Bun listener identity.
The native adapter test fails with the old drain; 20 passes per mode.
CI: all 24 integration shards and Next.js on Node 20/22/24 pass.
Executed gates and limits
Full core: 7,040 passing / 207 pending; LMDB: 3,256 / 722, at 5e3ec1e. RocksDB: 4,284 / 54; Bun: 1,961 passing, 32 skipped, one TODO, at 3cdf370.
Full Node: 2,391 passed; two tests cancelled after one installation timeout. Isolated baseline/current: 2/2 pass.
Final: Next.js 42; adapter 38 per mode; uWS runtime 22. Node/Bun runtime: 22/24 at 5e3ec1e. Build/types/lint/format/design pass.
Windows/Node22 units retain 357adfb failures; main's Windows job passed.
Package loading tests preserve ordinary loading and verify the ws entry.
Regression evidence
Regressions: typeStripSuites, preTestPrepNative, perFileDatabaseTeardown, branchDatabase, closeLoadedDatabases, longLivedTransactions, schema.test.js, pidInvariant.cjs, gitSSHMaterialization, storageExhaustion, rocksdbBackup.test.js, logConfigFixture.js.
🤔 Final review tradeoffs
Source-only omits compiled fixtures. Helper markers control lane selection. Local Mocha retains unbounded per-test waits; CI steps are bounded.
Review limits
Framing-Verdict: chosen-approach-sound
Review converged; 21-row trace passed. The diff was truncated. Gaps: handler completion #3261; Bun MQTT/TLS #3324. Configure before import; native-partition policy nit retained.
PR QA: Can read-only build children discover a live writer’s tables without changing storage? — queued @ f6ea41b, qa-pr-harper-3199-f6ea41b
Remaining test and fixture files
Paths: unitTests; bridge=dataLayer/harperBridge.
🤖 Generated by OpenAI Codex; posted via @kriszyp.
Release target: v5.4 — stacked on #562.
Related PRs: #562 overlaps (stacked foundation), #1835 overlaps (blob bindings), #2155 overlaps (eviction bindings), #3121 overlaps (backup fixtures), #3137 overlaps (MQTT tests), #3187 overlaps (handler aliases), #3323 overlaps (eviction bindings), #3282 overlaps (workflow pins)
Complexity: complicated — worker completion and module identity.
Deferred findings
1eada620d2e4e40dminor unclassified —.github/workflows/unit-test.yml— Native timeout caps lack hosted native measurements — filed Measure hosted native unit CI deadlines #319619f8b08d43209eaeminor performance —unitTests/typeStripSuites.mts— Shared helpers are reparsed separately for every suite — filed Measure repeated parsing in unit runtime selection #3197c163b5329a470bd8minor correctness —unitTests/mocha.init.js— Completion interception patches only project-installed Mocha — filed Track completion when a distinct Mocha installation runs units #31983a670ce90bdd04abminor reliability —server/mqtt.ts— MQTT socketless WebSockets bypass backpressure and closed-state detection — filed Verify Bun MQTT WebSocket flow control with the installed ws transport #3324451aea71a9a77983minor test-coverage —integrationTests/server/typestrip.test.ts— Producer cleanup does not prove REST send-loop or drain-wait completion — filed WebSocket disconnect never ends a connect() generator waiting on incomingMessages #326101111d5b8266a389nit maintainability —integrationTests/server/typestrip.test.ts— Added comment narrates adjacent stalled-producer assertions — filed Clarify the elapsed-time window in the stalled-producer test comment #3330Review-Coverage: authored=codex; ran=gemini,claude; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse,muse; rounds=14; full=1 @ f6ea41b
Review-Attention: deep ~35m (sensitive: unit-test.yml; hot: REST.ts, http.ts +1; decisions: do-less-alternative, eager-public-discovery, bun-websocket-backend, deferred-empty-drain, native-coverage-policy, import-time-discovery; raised: degraded review) @ f6ea41b