require('http') links runtime-only, so stdlib builtins return undefined (ESM import works)

[proggeramlug]

crates/perry/tests/issue_4903_listen_callback_deferred.rs fails both its tests on main, and it is now the only thing keeping the sweep-tier cargo-test red (and therefore main-gate).

thread 'late_listening_listener_fires' panicked at issue_4903_listen_callback_deferred.rs:45
thread 'listen_callback_is_deferred_and_this_bound' panicked at issue_4903_listen_callback_deferred.rs:45
test result: FAILED. 0 passed; 2 failed

Line 45 is the "compiled binary failed" assertion, so this is a runtime failure, not a wrong value.

Reduced. http.createServer returns undefined in every form, while the export itself is a function:

const http = require('http');
console.log('no-arg      :', typeof http.createServer());                       // undefined
console.log('with-handler:', typeof http.createServer(function () {}));         // undefined
console.log('arrow       :', typeof http.createServer(() => {}));               // undefined
console.log('createServer:', typeof http.createServer);                         // function

Node prints object for the first three. The test's own failure is downstream of this — .listen(0, cb) on undefined throws TypeError: Cannot read properties of undefined (reading 'listen').

Where it comes from. ("http", "createServer") in native_module_dispatch/dispatch_d_i.rs routes through JS_NATIVE_HTTP_DISPATCH, and returns undefined when that pointer is null:

let ptr = crate::value::JS_NATIVE_HTTP_DISPATCH.load(Ordering::SeqCst);
if ptr.is_null() { f64::from_bits(JSValue::undefined().bits()) } else { … }

That pointer is only installed by perry-stdlib's common/dispatch/init.rs:689, under #[cfg(feature = "external-http-server-pump")] — and that feature is not part of perry-stdlib's default/full set. So the question is whether the compile path that these tests exercise links a stdlib with the feature on. Locally the compile reports Linking (runtime-only), which is consistent with an http-using program being linked without the http implementation.

Two things I ruled out while narrowing, worth stating so the next person does not repeat them:

• It is not an artifact of my PERRY_RUNTIME_DIR override. I first reproduced with the override pointing at locally-built archives (which legitimately lack the opt-in feature), which would have been a build-config artifact; re-running with env -u PERRY_RUNTIME_DIR, exactly as the test does, reproduces identically.
• It is not new. The same two tests fail on c721cb6a0, 7c22189aa and e5ebb82b8, i.e. it predates today's merges; it only became the visible blocker once the other cargo-test failure (GC: defineProperty and Reflect/own-key paths use a stale receiver or value under forced evacuation #8507's suite, fixed by fix(gc): root defineProperty operands across exotic probes (#8507) #8526) and the two lint failures (fixed by fix(ci): convert new raw-handle debt to the rooting combinators (main lint is red) #8541) were cleared.

Repro: RUST_TEST_THREADS=1 cargo test --release -p perry --test issue_4903_listen_callback_deferred, or compile the four-line snippet above and run it.

[proggeramlug]

Root cause: the stdlib-linking decision sees ESM import but not require().

Same program, two import forms, everything else identical:

import * as http from 'node:http';   →  Linking (with stdlib)...   typeof createServer() === "object"   ✅
const http = require('http');        →  Linking (runtime-only)...  typeof createServer() === undefined  ❌

So this is not an http bug at all. require('http') does not mark the program as needing the stdlib, the link comes out runtime-only, perry-stdlib's common/dispatch/init.rs never runs, JS_NATIVE_HTTP_DISPATCH stays null, and the ("http", "createServer") arm in native_module_dispatch/dispatch_d_i.rs takes its documented null branch and returns undefined. Every symptom follows from that one fact.

The uses_dgram doc comment in commands/compile/types.rs points at the mechanism: native_module_imports "only tracks requires_stdlib modules", and it is populated from import statements. A CommonJS require("http") never lands in it.

That also explains the blast radius: any stdlib-backed builtin reached via require() rather than import is affected, not just http. The two failing tests in issue_4903_listen_callback_deferred.rs both use require('http'), which is why they fail while the ESM-based http coverage passes.

Fix direction: treat a require("<builtin>") call as a native-module import for the purposes of the requires_stdlib computation, so the link decision matches the ESM path. Worth checking whether the detection runs before or after any require → import normalisation, since that determines where the call sites are visible.

[proggeramlug]

Why the obvious fix does not work — the require path leaves no static module marker.

I tried recovering the requirement from the lowered HIR, the way uses_dgram does (hir_debug.contains("module: \"dgram\"")), scanning for any module: "X" where requires_stdlib(X) and setting needs_stdlib + native_module_imports. It builds, and it changes nothing: the link is still runtime-only. Reverted.

The reason is visible in --print-hir. For const http = require('http') the module name never becomes a module: marker at all — the only markers present are module: "__perry_runtime" and module: "module". What the CJS shim emits instead is a generated runtime dispatcher: a switch over builtin names compared as string literals.

SwitchCase { test: Some(String("node:fs/promises")), body: [] },
SwitchCase { test: Some(String("http")), body: [] },
SwitchCase { test: Some(String("node:http")) ... }
If { condition: Compare { op: Eq, left: LocalGet(22), right: String("http") }, ... }

So the name is resolved at runtime, and the HIR contains a case for every builtin regardless of what the program actually uses. A static scan for String("http") would match that table, not the call, and would link the stdlib into every CJS program.

That rules out the HIR as the detection point and pushes the fix to the source/AST level, where the require("<literal>") call site is still distinguishable. crates/perry/src/commands/compile/cjs_wrap/extract_requires.rs already extracts exactly those call sites for CJS wrapping, so it is the natural hook: feed literal-argument requires whose name satisfies perry_hir::requires_stdlib into ctx.native_module_imports / ctx.needs_stdlib alongside the existing import walk in collect_modules.rs.

Worth deciding deliberately: a dynamic require(someVar) cannot be resolved statically at all, and the generated dispatcher can reach any builtin, so someone needs to choose whether that case should force stdlib linking (correct, larger binaries) or keep returning undefined (current behaviour, silently wrong). That is a product call, not a mechanical fix, which is why I am leaving it here rather than guessing.

[proggeramlug]

Reopening: the fix in #8548 was reverted (#8551) because it caused a runtime-only link regression, even though the diagnosis below still stands.

What #8548 got right. require('http') never reached native_module_imports, so needs_stdlib stayed false, the link came out runtime-only, perry-stdlib's dispatch init never ran, JS_NATIVE_HTTP_DISPATCH stayed null, and the createServer arm returned undefined. Feeding literal require("<builtin>") specifiers (via the existing cjs_wrap::extract_require_specifiers) into the link decision fixes that: issue_4903_listen_callback_deferred went 0/2 → 2/2, and the scoped binary-size check showed the change was correctly targeted — a plain program and one using require("path") (a runtime-only module) both stayed runtime-only.

What it got wrong. Making a program need the stdlib is not free. issue_5247_property_read_source_location, which had been passing, started failing at link time:

undefined reference to `perry_wasm_host_validate'
undefined reference to `perry_wasm_host_module_new'
collect2: error: ld returned 1 exit status

perry-runtime/src/webassembly.rs declares those extern; only the separate perry-wasm-host crate defines them, and a runtime-only link line does not include it. Under cargo test --workspace cargo unifies features across the whole graph, so pulling perry-stdlib onto more compile paths changes what target/debug/libperry_runtime.a contains — and once the archive carries the wasm code, any runtime-only link against it fails.

Attribution, so nobody has to redo it: 5247 passed on 1c2326554 (the commit immediately before) and failed on b00e261f8 in both sweep runs of that SHA — deterministic. It does not reproduce locally, which is the signature of a feature-unification effect rather than a logic error; a local single-crate build simply does not produce the same archive.

So the remaining work is not the detection — it is the link-mode consequence. Whoever picks this up needs to decide one of:

1. put perry-wasm-host on the link line whenever the runtime archive contains wasm code, or
2. keep the wasm surface out of the runtime archive so a runtime-only link stays self-contained, or
3. gate the new stdlib requirement so it cannot change feature unification for unrelated compile paths.

Re-landing the detection without settling that will reproduce the same regression. The revert restores the known red (4903) rather than a new link-level one, since a link regression can bite real builds and not just tests.

[proggeramlug]

Correction — I attributed this to Cargo feature unification, and that was wrong.

I said the wasm link failure came from cargo test --workspace unifying perry-runtime/wasm-host on. That explanation does not survive checking: no workspace crate enables perry-runtime/wasm-host. The only occurrence in any Cargo.toml is the feature's own declaration (wasm-host = []). Unification cannot turn on a feature nothing requests.

The actual mechanism is shared-archive contamination between tests in the same job:

• optimized_libs/no_auto.rs builds perry-runtime-static with default features plus perry-runtime/wasm-host when ctx.needs_wasm_runtime is set — i.e. when the compiled program references WebAssembly.*. That is the PERRY_NO_AUTO_OPTIMIZE=1 path.
• issue_5247_property_read_source_location also runs with PERRY_NO_AUTO_OPTIMIZE=1 and points PERRY_RUNTIME_DIR at the same shared target/debug, rebuilding only -p perry-runtime (no wasm) via its own ensure_runtime_archive.

So under cargo test --workspace, any test that compiles a WebAssembly-using program rewrites the shared libperry_runtime.a with the wasm surface. A later runtime-only link from 5247 then hits undefined reference to perry_wasm_host_*, because the host archive is only put on the link line when that program sets needs_wasm_runtime. Whichever build wrote the archive last decides the outcome.

That reframes the situation in two ways worth being explicit about:

1. It is a pre-existing fragility, not something fix(compile): treat require(<builtin>) as a stdlib requirement (#8547) #8548 created. fix(compile): treat require(<builtin>) as a stdlib requirement (#8547) #8548 perturbed which tests rebuild the shared archive and in what order, which is why it flipped a latent race into a deterministic failure on that SHA. The revert is still correct — I would not leave a link-level red on a release branch — but "fix(compile): treat require(<builtin>) as a stdlib requirement (#8547) #8548 caused it" overstates my change's role, and I want that on the record before someone re-lands the detection and is surprised again.
2. The durable fix is archive isolation, not the link path. Two cargo invocations with different feature sets write the same target/debug/libperry_runtime.a; nothing arbitrates. Giving the runtime-archive-consuming tests their own PERRY_RUNTIME_DIR (or keying the archive path by feature set) removes the whole class. That is a smaller and safer change than either of the link-line options I listed earlier, and it would let require('http') links runtime-only, so stdlib builtins return undefined (ESM import works) #8547's detection re-land as-is.

I am not making that change myself: it touches test infrastructure I did not write, and I have already demonstrated on this issue that acting with an incomplete model of this link path produces regressions. Flagging it for an owner with the evidence above.