Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions docs/tests/report-test766-bunx-preflight.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
test766 — Bun package-runner preflight behavior gate
Date: 2026-08-13 Asia/Shanghai

Verdict: PASS

Coordinates
-----------
BASE_COMMIT=3b049160b2db542d06a32988a530f0b21f773c96
SOURCE_COMMIT=885a74d7c5418d0ee20de46c728396a367ec41cf
IMAGE_ID=sha256:60994c0f0a088167e7f5ed77f74fc5fefbde57be414bd9a23b1764f48dc746bc
FOCUSED_RUN_LOG_SHA256=1779a95c74afdd17670d2311dc99f09605b671ff1ecfc8dac9a209c4b0de55f8
L1_RUN_LOG_SHA256=fca1054ab0b7604a0d86e2a1f212e7d78b7214c2702c53e0e4d4a62d7b310a03

Scope
-----
- Test/CI only: no product, package, dist, deployment, database, or runtime
configuration changes.
- Adds test766 to the active scripts/qa.sh L1 list.
- Pins the behavior selected by #767; it does not decide whether #766 should
later adopt the separately tested automatic `bun x` fallback design.

Focused Docker result
---------------------
The image used Bun 1.3.14 pinned by digest:
oven/bun:1.3.14@sha256:e10577f0db68676a7024391c6e5cb4b879ebd17188ab750cf10024a6d700e5c4

Observed output:
source_commit=885a74d7c5418d0ee20de46c728396a367ec41cf
PASS bun-only, neither, and bunx behavior matrix
PASS agent-network production build
PASS mutation old-or-allows-bun-only witnessed red at its named behavior
PASS mutation valid-bunx-blocked witnessed red at its named behavior
RESULT pass=4 fail=0

Behavior matrix
---------------
1. `bun` without `bunx`: nonzero before spawn, precise bun-only diagnosis and
manual symlink remediation, with no healthy-start banner.
2. Neither executable: nonzero with the generic Bun installation guidance,
without the bun-only diagnosis or healthy-start banner.
3. `bunx` available: the production CLI crosses the preflight, passes exact
argv `--bun @sleep2agi/commhub-server@0.9.0-preview.29`, and reaches the
healthy-Hub banner against the fixture server.

Witnessed-red mutations
-----------------------
- `old-or-allows-bun-only`: restores the former permissive
`!bunx && !bun` guard. The clean bun-only probe is green; the mutation reaches
spawn and fails the named preflight behavior.
- `valid-bunx-blocked`: forces the guard to reject a valid bunx installation.
The clean bunx probe is green; the mutation prevents runner capture and the
healthy banner, so the named behavior turns red.
- Each mutation first requires an exact single production target and confirms
the changed text. A missing or duplicated target fails before classification.

Active L1 regression
--------------------
Command:
bash scripts/qa.sh --l1

Result on SOURCE_COMMIT:
15/15 active L1 suites passed
test766: RESULT pass=4 fail=0
ALL PASS in 107s
measured wall time: 107.24s

Provenance
----------
- The focused image recorded TEST766_SOURCE_COMMIT equal to SOURCE_COMMIT.
- The six files copied into the focused image under
tests/test766-bunx-preflight were compared byte-for-byte with SOURCE_COMMIT:
PROVENANCE total=6 fail=0.
- scripts/qa.sh is exercised by the host L1 runner and is not copied into the
focused image; its inclusion of test766 is proven by the 15-suite output.

Honest limits
-------------
- The healthy path uses an HTTP fixture after the real CLI has selected and
invoked bunx. It proves preflight and argv wiring, not registry download or
the full commhub-server implementation; existing Hub E2E suites remain the
authority for those behaviors.
- This report does not claim a production rollout, npm publication, or a final
product decision for bun-only installations.
3 changes: 3 additions & 0 deletions scripts/qa.sh
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ L1_TESTS=(
"qa-node-02-success-reply"
"qa-node-03b-task-events"
"test686-rest-shape-golden"
"test766-bunx-preflight"
)

if [[ "${1:-}" == "--list" ]]; then
Expand Down Expand Up @@ -143,6 +144,8 @@ if [[ $RUN_L1 -eq 1 ]]; then
build_args=""
if [[ "$t" == "test686-rest-shape-golden" ]]; then
build_args="--build-arg TEST686_SOURCE_COMMIT=$(git rev-parse HEAD)"
elif [[ "$t" == "test766-bunx-preflight" ]]; then
build_args="--build-arg TEST766_SOURCE_COMMIT=$(git rev-parse HEAD)"
fi
if ! dockerrun "docker build -q $build_args -t anet-$t -f tests/$t/Dockerfile ." >/tmp/qa-l1-$t-build.log 2>&1; then
fail "L1 $t — build failed, see /tmp/qa-l1-$t-build.log"
Expand Down
17 changes: 17 additions & 0 deletions tests/test766-bunx-preflight/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
FROM oven/bun:1.3.14@sha256:e10577f0db68676a7024391c6e5cb4b879ebd17188ab750cf10024a6d700e5c4

WORKDIR /repo
ARG TEST766_SOURCE_COMMIT=unknown
ENV TEST766_SOURCE_COMMIT=$TEST766_SOURCE_COMMIT

COPY agent-network/package.json /repo/agent-network/package.json
RUN cd /repo/agent-network && bun install --ignore-scripts
Comment on lines +7 to +8

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Copy the lockfile before running bun install

Because only package.json is present when bun install runs, the image resolves every ranged dependency from the registry instead of using the committed agent-network/package-lock.json; copying the repository afterward is too late. A new compatible dependency release can therefore make this active L1 suite pass or fail differently for the same source commit, undermining the recorded reproducibility guarantees. Copy the lockfile before installation and use a frozen-lockfile install.

AGENTS.md reference: AGENTS.md:L25-L25

Useful? React with 👍 / 👎.


COPY agent-network /repo/agent-network

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude host node_modules from the Docker context

When a checkout contains the ignored agent-network/node_modules directory, as it normally does after local setup, this COPY merges those host files over the dependencies installed in the preceding image layer because Docker does not honor .gitignore and this repository has no .dockerignore. The supposedly isolated suite can therefore use stale or host-specific dependency artifacts and produce results that differ between otherwise identical Docker runs; exclude node_modules from the build context or copy only tracked source paths.

AGENTS.md reference: AGENTS.md:L7-L7

Useful? React with 👍 / 👎.

COPY tests/lib /repo/tests/lib
COPY tests/test766-bunx-preflight /repo/tests/test766-bunx-preflight
RUN chmod +x /repo/tests/test766-bunx-preflight/run.sh \
/repo/tests/test766-bunx-preflight/fake-bun-only/bun \
/repo/tests/test766-bunx-preflight/fake-bunx/bunx

CMD ["bash", "/repo/tests/test766-bunx-preflight/run.sh"]
16 changes: 16 additions & 0 deletions tests/test766-bunx-preflight/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# test766 — Bun package-runner preflight

Commits the three behavioral cases chosen by #767:

1. `bun` exists and `bunx` does not: fail before spawn with the precise
remediation message;
2. neither exists: retain the generic Bun installation message;
3. `bunx` exists: cross the preflight, preserve the exact package argv, and
reach the CLI's healthy-Hub banner.

Two witnessed-red mutations restore the old permissive OR and make the guard
reject a valid bunx installation. This suite is listed in `scripts/qa.sh` L1;
it is not an uncalled local probe.

The healthy-path HTTP fixture covers CLI wiring and does not claim to replace
the repository's real commhub-server E2E suites.
2 changes: 2 additions & 0 deletions tests/test766-bunx-preflight/fake-bun-only/bun
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/sh
exit 0
6 changes: 6 additions & 0 deletions tests/test766-bunx-preflight/fake-bunx/bunx
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/sh
set -eu

capture=${TEST766_CAPTURE:?TEST766_CAPTURE is required}
printf '%s\n' "$@" > "$capture"
exec /usr/local/bin/bun /repo/tests/test766-bunx-preflight/fixture-server.ts "$@"
35 changes: 35 additions & 0 deletions tests/test766-bunx-preflight/fixture-server.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
const port = Number(process.env.PORT || "27668");
const capture = process.env.TEST766_CAPTURE;
if (!capture) throw new Error("TEST766_CAPTURE is required");

const server = Bun.serve({
hostname: "127.0.0.1",
port,
fetch(request) {
const path = new URL(request.url).pathname;
if (path === "/health") {
return Response.json({ ok: true, version: "test766-bunx" });
}
if (path === "/api/auth/login") {
return Response.json({ ok: false, error: "invalid credentials" });
}
if (path === "/api/auth/register") {
return Response.json({
ok: true,
token: "utok_test766_fixture",
user: { user_id: "user-766", username: "admin" },
});
}
return Response.json({ ok: false, error: "not found" }, { status: 404 });
},
});

await Bun.write(`${capture}.pid`, String(process.pid));

function stop() {
server.stop(true);
process.exit(0);
}
process.on("SIGTERM", stop);
process.on("SIGINT", stop);
await new Promise(() => {});
142 changes: 142 additions & 0 deletions tests/test766-bunx-preflight/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
#!/usr/bin/env bash
set -Eeuo pipefail

ROOT=/repo
ART=/artifacts
PASS=0
FAIL=0
mkdir -p "$ART"
source "$ROOT/tests/lib/safe-rm.sh"

ok(){ PASS=$((PASS+1)); printf 'PASS %s\n' "$*"; }
bad(){ FAIL=$((FAIL+1)); printf 'FAIL %s\n' "$*"; }

probe_bun_only(){
local case_root home log rc
case_root=$(mktemp -d /tmp/test766-bun-only.XXXXXX)
home="$case_root/home"
log="$case_root/cli.log"
mkdir -p "$home"
set +e
HOME="$home" PATH="$ROOT/tests/test766-bunx-preflight/fake-bun-only:/usr/bin:/bin" \
/usr/local/bin/bun "$ROOT/agent-network/bin/cli.ts" hub start \
--host 127.0.0.1 --port 27668 --username admin --password StrongPassw0rd \
>"$log" 2>&1
rc=$?
set -e
if [[ "$rc" -eq 0 ]] \
|| ! grep -Fq '找到了 bun,但没有 bunx' "$log" \
|| ! grep -Fq 'ln -s "$(command -v bun)"' "$log" \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check the complete bunx symlink remediation

The bun-only probe matches only the source half of the suggested ln -s command. If the destination is removed or changed to an invalid path, the suite still passes even though the user-facing remediation it explicitly intends to preserve no longer creates bunx. Assert the complete command, including "$(dirname "$(command -v bun)")/bunx", rather than this prefix.

Useful? React with 👍 / 👎.

|| grep -Fq 'Starting CommHub Server' "$log"; then
cat "$log" >&2
safe_rm_rf "$case_root"
return 1
fi
safe_rm_rf "$case_root"
}

probe_neither(){
local case_root home log rc
case_root=$(mktemp -d /tmp/test766-neither.XXXXXX)
home="$case_root/home"
log="$case_root/cli.log"
mkdir -p "$home"
set +e
HOME="$home" PATH="/usr/bin:/bin" \
/usr/local/bin/bun "$ROOT/agent-network/bin/cli.ts" hub start \
--host 127.0.0.1 --port 27668 --username admin --password StrongPassw0rd \
>"$log" 2>&1
rc=$?
set -e
if [[ "$rc" -eq 0 ]] \
|| ! grep -Fq 'anet hub start requires the Bun runtime' "$log" \
|| grep -Fq '找到了 bun,但没有 bunx' "$log" \
|| grep -Fq 'Starting CommHub Server' "$log"; then
cat "$log" >&2
safe_rm_rf "$case_root"
return 1
fi
safe_rm_rf "$case_root"
}

probe_bunx(){
local case_root home capture log cli_pid fixture_pid rc
case_root=$(mktemp -d /tmp/test766-bunx.XXXXXX)
home="$case_root/home"
capture="$case_root/runner.argv"
log="$case_root/cli.log"
mkdir -p "$home"

HOME="$home" TEST766_CAPTURE="$capture" \
PATH="$ROOT/tests/test766-bunx-preflight/fake-bunx:/usr/bin:/bin" \
/usr/local/bin/bun "$ROOT/agent-network/bin/cli.ts" hub start \
--host 127.0.0.1 --port 27668 --username admin --password StrongPassw0rd \
>"$log" 2>&1 &
cli_pid=$!

for _ in $(seq 1 100); do
[[ -s "$capture" ]] \
&& grep -Fq 'Server running on http://127.0.0.1:27668' "$log" \
&& break
sleep 0.05
done

rc=0
[[ -s "$capture" ]] || rc=1
[[ "$(sed -n '1p' "$capture" 2>/dev/null || true)" == "--bun" ]] || rc=1
grep -Fxq '@sleep2agi/commhub-server@0.9.0-preview.29' "$capture" || rc=1
Comment on lines +86 to +87

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert the complete bunx argument vector

The healthy-path probe only requires --bun to be first and the package string to occur somewhere in the capture. If the CLI appends an unintended argument, including a flag that changes server behavior, the fixture ignores it and this suite still passes despite claiming to preserve the exact argv. Compare the entire capture, including its line count and order, against the expected two arguments.

Useful? React with 👍 / 👎.

grep -Fq 'Server running on http://127.0.0.1:27668' "$log" || rc=1

kill "$cli_pid" 2>/dev/null || true
fixture_pid=$(cat "$capture.pid" 2>/dev/null || true)
[[ -n "$fixture_pid" ]] && kill "$fixture_pid" 2>/dev/null || true
wait "$cli_pid" 2>/dev/null || true
if [[ "$rc" -ne 0 ]]; then
cat "$log" >&2 || true
sed -n '1,20p' "$capture" >&2 || true
safe_rm_rf "$case_root"
return 1
fi
safe_rm_rf "$case_root"
}

probe_all(){
probe_bun_only
probe_neither
probe_bunx
}

expect_red(){
local name="$1"; shift
if "$@" >"$ART/$name.log" 2>&1; then
bad "mutation $name survived"
else
tail -50 "$ART/$name.log"
ok "mutation $name witnessed red at its named behavior"
fi
}

printf 'source_commit=%s\n' "${TEST766_SOURCE_COMMIT:-unknown}"
cd "$ROOT"

probe_all
ok "bun-only, neither, and bunx behavior matrix"
(cd agent-network && bun run build)
ok "agent-network production build"

cp agent-network/bin/cli.ts /tmp/test766-cli.orig

[[ "$(grep -Fc 'if (!commandExists("bunx")) {' agent-network/bin/cli.ts)" -eq 1 ]]
sed -i 's/if (!commandExists("bunx")) {/if (!commandExists("bunx") \&\& !commandExists("bun")) {/' agent-network/bin/cli.ts
grep -Fq 'if (!commandExists("bunx") && !commandExists("bun")) {' agent-network/bin/cli.ts
expect_red old-or-allows-bun-only probe_bun_only
cp /tmp/test766-cli.orig agent-network/bin/cli.ts

[[ "$(grep -Fc 'if (!commandExists("bunx")) {' agent-network/bin/cli.ts)" -eq 1 ]]
sed -i 's/if (!commandExists("bunx")) {/if (true || !commandExists("bunx")) {/' agent-network/bin/cli.ts
grep -Fq 'if (true || !commandExists("bunx")) {' agent-network/bin/cli.ts
expect_red valid-bunx-blocked probe_bunx
cp /tmp/test766-cli.orig agent-network/bin/cli.ts

printf 'RESULT pass=%s fail=%s\n' "$PASS" "$FAIL"
[[ "$FAIL" -eq 0 ]]
Loading