ci-local's VM target dirs are ~39 GB each, and 96% of that is debug info

[TinDang97]

Measured

scripts/ci-local.sh builds two VM-local cargo target dirs (monoio + tokio legs). Each is ~39 GB, so the pair is ~75 GB on moon-dev's 111 GB root. Measured 2026-08-22:

~/ci-target/local-monoio/          39 GB
  debug/deps/                      32 GB
    238 test executables           30.5 GB   avg 128 MB, largest 251 MB
    297 rlibs (the actual deps)     1.0 GB
  debug/incremental/                4.9 GB

tests/ holds 233 integration test files. Each is its own crate statically linked against the whole moon lib plus every dependency, so the dep graph is duplicated 233 times.

~96% of each test binary is DWARF. One 191 MB test binary drops to 8 MB under strip --strip-debug. So of the 30.5 GB of test executables, roughly 29 GB is debug info and ~1.5 GB is code.

Why it matters

The failure mode is silent and reads as a code failure. When the VM root fills, OrbStack wedges: the leg exits rc=1 with no error text (the log shows only Compiling moon), orb run then answers sconrpc ready event fired but socket was not connectible, and orb list shows the machine stopping. This happened on PR #654 — the monoio leg had already passed 5885/5885 and the tokio leg died mid-compile purely from disk pressure.

Proposed fix

[profile.dev]
debug = "line-tables-only"

Panic messages carry file:line from #[track_caller] regardless of this setting, and line tables keep backtraces readable. Nothing in these suites is run under a debugger. Expected effect, from the strip measurement: ~30 GB → single-digit GB per target dir, and proportionally faster linking.

Worth checking as part of the change:

• a panicking test still reports a useful backtrace with RUST_BACKTRACE=1
• the crash-matrix suites, which inspect core dumps / stack traces if any do
• wall-clock delta on a cold cargo nextest run (link time should drop)

Also worth considering

238 test binaries from 233 files is a lot of duplicated linking. Consolidating related suites into fewer crates would compound the win, but that is a larger refactor and should not block the one-line profile change.

[TinDang97]

This bit twice in one session today (2026-08-24), and the failure mode is worse than slow — it
is a CI leg that fails with a linker error and looks like a code problem:

/usr/bin/aarch64-linux-gnu-ld.bfd: final link failed: No space left on device
error: could not compile `moon` (test "tls_idle_downshift_parity")
error: could not compile `moon` (test "graph_restart_id_aliasing")
✗ VM monoio suite (shipped runtime) FAILED (rc=101)

Nothing in that output says "disk". The two named test targets are arbitrary — whichever
happened to link when the device filled — so the obvious next move is to go and read
tls_idle_downshift_parity, which has nothing to do with anything.

Measured

Before the run: 20 GB free. During it, local-monoio grew 38 GB → 57 GB, and the two
legs together reached 92 GB of a 129 GB disk:

1.4G  ci-target/client-compat
4.5G  ci-target/check-monoio
 35G  ci-target/local-tokio
 57G  ci-target/local-monoio     <- 38G at the start of this run

df went 84% → 98% mid-run. Deleting both legs recovered it to 27%, at the cost of a full
rebuild of each.

So the ~39 GB in the title is the steady state; a single suite build can add another 19 GB on
top before anything is reclaimed. The headroom needed is not "39 GB per leg", it is closer to
"60 GB per leg, transiently".

Suggestions, in order of how cheap they are

1. Fail loudly and early. ci-local.sh already has a host-volume pre-flight (ci-local disk pre-flight passes while the host volume backing the VM is full #661/fix(ci): pre-flight the HOST volume behind the VM, not just the VM (#661) #675);
   the VM legs need the same, checking free space against a realistic transient figure rather
   than the steady-state one. A leg that refuses to start with "need ~60 GB free, have 20 GB"
   costs seconds; one that dies at link time costs the whole build plus the misdiagnosis.
2. Stop generating what nobody reads. 96% debug info, per the title. A [profile.test]
   with debug = "line-tables-only" keeps backtraces useful and should cut the bulk of it.
   Worth measuring before committing to it.
3. Budget the cache. Prune legs by age or total size at the start of a run rather than
   letting two 57 GB trees coexist indefinitely.

Also relevant: in-guest rm frees space in the guest but nothing on the host until fstrim,
so anyone reasoning about host headroom needs to trim before re-reading df.

[TinDang97]

Second measurement, 2026-08-24 (moon-dev, after the #696 dispatch matrix and one ci-local run):

37G  ~/ci-target/local-monoio
35G  ~/ci-target/local-tokio
5.7G ~/ci-target/check-monoio
1.5G ~/ci-target/client-compat
---
78G  total, on a 128G root -> 22G free (83% used)

22G is below what a single leg needs transiently (the earlier run grew local-monoio 38G -> 57G during the run), so ci-local could not be started without pruning first. Deleting both local legs took the root back to 29% used and cost a full cold rebuild of both.

Two operational notes worth folding into any fix:

1. df under-reports immediately after the rm -rf. The first df after deleting 72G showed only 13G reclaimed; the unlink is still in flight. A prune step that checks its own result needs a sync or a re-read, or it will conclude the prune failed and either bail or delete more than it meant to.
2. The failure mode when you don't prune is not a disk error. It surfaces as No space left on device at link time naming two arbitrary test targets (tls_idle_downshift_parity, graph_restart_id_aliasing last time) with nothing in the message pointing at disk — see the earlier comment. That is what makes this worth fixing rather than living with.

Given 96% of the bytes are debug info, the cheapest fix is still debug = 0 (or debug = "line-tables-only") for the profile ci-local uses. The legs run tests and assert exit codes; they do not read backtraces symbolically, and the one thing that would want symbols — a panic backtrace — keeps its file/line with line-tables-only.

[TinDang97]

Two corrections to the disk pre-flight's own guidance, found by hitting the guard today.

1. The "fstrim returns almost nothing" advice is wrong — the reclaim is asynchronous.

scripts/ci-local.sh currently tells the operator:

Measured 2026-08-23: 82.8 GiB reported trimmed returned under 1G of host space, because most of it was already sparse.

That reading was taken too soon. Today, after rm -rf ~/ci-target/{local-tokio,local-monoio} (78 G) in the guest:

df -h /            -> 11Gi avail        (before fstrim)
sudo fstrim -av    -> 345.4 GiB trimmed
df -h /            -> 11Gi avail        (immediately after — looks like a no-op)
df -h / (later)    -> 32Gi avail        (~21G actually returned)

APFS returns the blocks asynchronously, so the df immediately after fstrim is worthless as a verdict. As written, the message tells an operator the reclaim failed at exactly the moment it is succeeding — which invites deleting more caches, and rebuilding 78 G of target dirs that did not need deleting.

Suggest replacing that paragraph with "re-check after a minute; APFS returns the blocks asynchronously and the immediate reading will look unchanged."

2. The guard reads df /, which on macOS is the sealed read-only system volume.

The real figure lives on /System/Volumes/Data:

/dev/disk3s3s1   460Gi    10Gi    32Gi    25%   /
/dev/disk3s1     460Gi   395Gi    32Gi    93%   /System/Volumes/Data

They agree on available here because both draw from the same APFS container free space, so the guard is not currently producing wrong verdicts. But used differs by 385 G, so any future change that reasons about used/total rather than avail would be silently wrong. Worth pinning to the Data volume while this is being touched anyway.

Context: the guard itself did its job — it refused to start and named the two 40 G/38 G consumers, which is exactly the behaviour that keeps a run from wedging OrbStack. Only the remediation advice is off.

[TinDang97]

Correction to point 2 of my previous comment — it is wrong, and I should have read the function before claiming it.

The guard does not read df /. scripts/ci-local.sh:138 already pins the Data volume, and read_host_avail_bytes documents the exact trap I "reported":

HOST_VOLUME=/System/Volumes/Data   # NOT `/`: see read_host_avail_bytes

That was fixed in #675 ("pre-flight the HOST volume behind the VM"), which predates my comment. There is nothing to change there; disregard that half.

Point 1 (the fstrim advice being read too early) still stands, and today gave it a second independent measurement:

df -h /System/Volumes/Data   -> 12Gi avail    (before)
orb run -m moon-dev sudo fstrim -av  -> 71.5 GiB trimmed
df -h /System/Volumes/Data   -> 13Gi avail    (immediately after — looks like a no-op)
df -h /System/Volumes/Data   -> 21Gi avail    (minutes later)

Same shape as the 2026-08-23 reading the script quotes: the immediate df under-reports by ~8G here, and the script's baked-in "returned under 1G of host space" is that same too-early reading recorded as fact. An operator who trusts it deletes more caches than they need to.

One addition worth folding in when this is touched: reclaim the secondary ~/ci-target/* dirs (check-monoio, crash-matrix, bench513) before the two big ones. Deleting local-monoio/local-tokio makes both ci-local legs cold, and vm_growth_bytes then projects 36G per cold leg instead of 2G warm — so the host verdict gets worse, not better, which is the opposite of what the remediation text is steering toward.

[TinDang97]

Measured the proposed change on moon-dev (aarch64 Linux, rustc 1.94.1) before implementing it. The proposal holds, and by more than the issue estimated.

Current state re-confirmed live (2026-08-26):

/dev/vdb1  159G  118G   42G  74% /
  ~/ci-target/local-monoio   42G
  ~/ci-target/local-tokio    40G

So 82G of a 159G root, not the 75G/111G recorded in August — the pair has grown.

A/B on one integration-test binary (--test ft_search_missing_index_728, cold target dir per leg, artifact path taken from cargo build --message-format=json rather than a glob):

	test binary	debug/moon	whole target dir
debug = 2 (today)	181,855,288 (173.4 MB)	256,789,664	3,309,719,552
debug = "line-tables-only"	56,483,152 (53.9 MB)	111,611,368	2,311,974,912
delta	−68.9%	−56.5%	−30.1%

The DWARF share is confirmed independently: strip --strip-debug takes that same 173.4 MB binary to 4,401,896 bytes (4.2 MB) — 97.6% debug info, matching the issue's measurement.

Extrapolating −68.9% across the ~30.5 GB of test executables in a leg puts them near 9.5 GB, i.e. roughly 20 GB reclaimed per leg / 40 GB across the pair.

Checklist items:

• a panicking test still reports a useful backtrace — verified on both settings. The panic header and every RUST_BACKTRACE=1 frame keep file:line (panicked at tests/…rs:5:5, frames at ./tests/…rs:5:5). Identical output under both, which is the expected result: line tables are exactly what a backtrace resolves against.
• the binary still runs — both legs list and execute their tests.
• crash-matrix suites that inspect core dumps / stack traces — there are none. grep across scripts/, .github/, tests/, src/ finds no reference to addr2line, gdb, lldb, coredump, or core_pattern, and the backtrace crate is not a dependency. Nothing consumes full DWARF.
• wall-clock delta — not isolated yet; the two legs above ran 1m28s vs 58s cold, but that pair is one build each and not a controlled timing measurement, so I am not claiming a link-time number from it.

One thing the change must carry with it: scripts/ci-local.sh hardcodes the current sizes —
DISK_COLD_NEED_BYTES=36G, DISK_WARM_WARN_BYTES=15G, DISK_COLD_DIR_BYTES=5G ("a smaller dir is a stub, not a cache"), and DISK_WARM_GROWTH_BYTES. After the profile change a healthy cold leg no longer needs 36G and a healthy warm dir may fall under the 5G stub threshold, so the pre-flight would start mis-judging healthy runs. Its own comment names that failure mode: "A pre-flight that grounds healthy runs gets disabled, and then it guards nothing." The constants get re-measured and updated in the same PR.

Caveat for anyone reproducing this on macOS: don't. Mach-O keeps DWARF in the .o/.dSYM files rather than the executable, so both settings produce a byte-identical test binary on the host and the change looks like a no-op. The effect is only visible on ELF.