test(installer): contract lane for built packages on a pristine apk-era VM — #888/#890 acceptance, shipped red

changelog.d/892-installer-contract-lane.internal.md

@@ -0,0 +1,26 @@
+- **The installer has its own fail-closed acceptance gate.** A built `.apk`
+  can be validated end-to-end on a pristine OpenWrt 25.12 VM exactly the
+  way an operator installs it — including the
+  `TOLLGATE_ADMIN_PASSWORD` / `TOLLGATE_PRIVATE_WIFI_PASSWORD`
+  environment the docs promise — with assertions for the credential
+  lifecycle (sha256crypt comparison, no plaintext in output, the
+  provisional/first-run fallback), services, listeners, the entry-port
+  mapping and reboot persistence, plus serial bootstrap and distrusted
+  pristine verification (a false `firstboot` "reset" on one 25.12.5 image
+  kept `/etc` and would have silently invalidated every credential
+  check). The gate is fail-closed end to end: an unavailable VM or an
+  unexecuted required check reports `NOT_VERIFIED` and exits nonzero
+  (never a silent PASS or SKIP), `--source-sha` binds the run to the
+  commit the artifact was built from, every run writes a machine-readable
+  `verdict.json` (per-check statuses, artifact sha256, image identity,
+  lane environment), `uci show uhttpd` is inspected so a listenerless
+  section is caught even when another instance claims its ports (#899),
+  crash-loop signatures (`No sockets bound`, segfaults) are scanned
+  before and after a reboot, and an offline selftest plants the #899 and
+  #911 regressions to prove the checks can go red. Known-reds ship on
+  purpose as acceptance criteria: #888 (the env never reaching the
+  uci-defaults script under `apk add`), #890 (the entry-port flip's LuCI
+  pair not converging on fresh-image installs) and #911 (apk-tools 25.12
+  completing the transaction — package registered, service started —
+  despite the failing preinst, reported verbatim)
+  ([#892](https://github.com/OpenTollGate/tollgate-module-basic-go/pull/892)).

tests/installer-contract/README.md

@@ -0,0 +1,153 @@
+# Installer-contract lane (built package → pristine apk-era VM)
+
+Asserts the **installer's** contracts against a live OpenWrt 25.12 (apk-era)
+device, using a built `.apk` and the documented operator flow — including the
+install-time credential environment (`TOLLGATE_ADMIN_PASSWORD`,
+`TOLLGATE_PRIVATE_WIFI_PASSWORD`). It complements
+[tests/happy-path/](../happy-path/) (which exercises the artifact bytes on the
+lane host) by covering what only a real install can: apk dependency closure,
+uci-defaults execution, credential lifecycle, listener/mapping convergence and
+reboot persistence.
+
+Origin: every check here was first verified by hand on the vlab lane on
+2026-10-11 (OpenWrt 25.12.5 x86-64, `tollgate-wrt-0.6.0_rc1-r0.apk`), which is
+how #888 was found. This script is that session, repeatable.
+
+## Run it
+
+```bash
+bash tests/installer-contract/run.sh \
+  --artifact ~/rel-artifacts/apk-x86-64/tollgate-wrt-…apk \
+  --source-sha "$(git rev-parse HEAD)" \
+  --host 127.0.0.1 --port 2222 --ssh-key ~/.ssh/tg_vlab_ed \
+  --serial 127.0.0.1:4554 \
+  --out ~/installer-contract.out
+```
+
+Prerequisites:
+
+- a **pristine** OpenWrt 25.12 VM (the lane runs `firstboot -y` itself unless
+  `--skip-pristine`) with **key-based root ssh** already working — or a serial
+  console (`--serial`, qemu `-serial telnet:…`) so the lane can bootstrap the
+  dhcp fix and its key itself after the reset;
+- `openssl` on the lane host (credential comparison runs host-side — the
+  device's busybox has no crypt tool for this);
+- the built `.apk`, and the commit it was built from (`--source-sha`; without
+  it, `provenance:source-sha` reports NOT_VERIFIED and the run fails).
+
+Exit 0 = all contracts hold. Output idiom matches the happy-path suite:
+
+```
+ICCHECK <id> <PASS|FAIL|SKIP|NOT_VERIFIED> <detail...>
+ICRESULT total=N pass=N fail=N skip=N not_verified=N
+ICFAILED <ids...>
+ICEXIT 0|1
+```
+
+**Fail-closed semantics.** `NOT_VERIFIED` is a first-class, failing status: a
+required check whose input could not be obtained — no VM reachable, a phase
+skipped without `--dev`, provenance not recorded — is never reported as PASS
+or silently skipped. `--skip-pristine`/`--skip-reboot` are dev-only escapes;
+without `--dev` they fail the run as NOT_VERIFIED. Every run also writes
+`verdict.json` into the evidence dir: per-check statuses with details, the
+artifact's sha256, the recorded source SHA, the image identity
+(`/etc/openwrt_release`, kernel, apk version) and the lane host environment —
+plus `refuse.log` / `install.log` (raw apk output) and `checks.tsv`.
+
+## Prove the gate can fail, before spending a VM on it
+
+```bash
+bash tests/installer-contract/selftest.sh
+```
+
+Runs the exact check logic (`checks.sh`, shared with the live lane) against
+planted fixtures, no VM and no network: the #899 tear (a listenerless
+`uhttpd.main`), including the stealth variant where another instance claims
+the ports so netstat liveness alone would read green; the #911 shape
+(apk-tools completing the transaction — package registered, service started,
+exit 1, guidance suppressed — despite the failing preinst); mapping
+mismatches; and crash-loop log signatures. Healthy fixtures stay green, so
+the controls are not tautologies. `SELFTESTRESULT total=18 ok=18 bad=0`,
+exit 0.
+
+## The rc3 contract (and the remaining known-red)
+
+Since #902 + #907 (v0.6.0-rc3) the package **sets no credential at all**
+and the **preinst refuses any install while root has no password**. The
+lane asserts exactly that:
+
+- `contract:preinst-refuses-empty-root` — a pristine install attempt must
+  fail cleanly with the guidance message and register nothing;
+- `flow:operator-sets-password` + `contract:package-sets-no-credential` —
+  the operator sets the root password, the install succeeds, and root's
+  hash is byte-identical across the install (nothing applied, replaced or
+  generated); `contract:no-provisional-marker` pins the same on disk;
+- persistence: the operator's credential and the mapping survive a reboot.
+
+Earlier revisions of this lane asserted the #894 staged-secret channel;
+that mechanism was absorbed by #897 and then removed by #902's stronger
+design — the history lives in #888.
+
+The remaining known-reds are kept asserted on purpose:
+
+- **#888's** `contract:credential-applied` (apk-tools does not deliver the
+  install-time environment to the uci-defaults script);
+- **#890's** `listeners:8090` / `listeners:8443` and, reading the config
+  itself rather than the wire, `listeners:uhttpd-sections` /
+  `mapping:port-ownership` — the #899 tear (a listenerless `uhttpd.main`)
+  still present on fresh-image installs;
+- **#911's** `contract:preinst-refuses-empty-root` — apk-tools 25.12
+  completes the transaction (package registered, service started, exit 1,
+  guidance suppressed) despite the failing preinst. The check reports the
+  observed shape verbatim and goes green only when apk actually refuses.
+
+A fence that cannot fail is decoration; the reds are shipped on purpose.
+
+## The pristine phase trusts nothing
+
+On one 25.12.5 x86 image `firstboot -y && reboot` silently kept `/etc`
+(dropbear keys, an installed tollgate-wrt, root's old hash) — a false
+"pristine" that turns every credential assertion into a lie. The lane
+therefore (a) forces `reboot -f`, and (b) **verifies** the clean state
+(no tollgate-wrt registered, root credential empty) before installing.
+For a snapshot-mode VM, the true reset is restarting the VM process
+(qemu `snapshot=on` discards the volatile disk); labgrid release/acquire
+is lock-only on this provider and does not restart it.
+
+## vlab notes (lessons already paid for)
+
+- The qemu user-net VM ships with `br-lan` static (`192.168.1.1`, no default
+  route): `uci set network.lan.proto='dhcp' && uci commit network &&
+  /etc/init.d/network restart` gives it `10.0.2.15` via slirp, and only then
+  do hostfwd ssh and outbound feeds work.
+- Install a lane key via the serial console (`echo <pubkey> >
+  /etc/dropbear/authorized_keys`); `scp` needs `-O` or a `cat`-pipe — the
+  guest's dropbear has no sftp-server.
+- VM host keys churn with snapshot reprovisioning: the lane already passes
+  `UserKnownHostsFile=/dev/null`.
+- **One user-net NIC only.** A VM given two `-netdev user` NICs (the vm-campaign
+  shape) ends up with the wan side *and* `br-lan` both holding `10.0.2.15`
+  after the dhcp reprovision — the duplicate address breaks `hostfwd` (ssh
+  banner timeout) in ways that look like a provisioning failure. This lane's
+  bring-up assumes the single-NIC vlab shape; the wan-routing need the second
+  NIC served is moot once `lan.proto=dhcp`.
+- The x86 VM has **no radios and no br-private**: SSID derivation, private
+  networking and band steering need the physical lane (PRTA journey on the
+  x1860 seats).
+
+## What is out of scope here (what a green run does NOT prove)
+
+Radios/SSID, br-private flows, real captivity against NoDogSplash on hardware,
+ipk/opkg-era installs (the 24.10 frozen lane), and the portal SPA behavior
+(happy-path's ground). Beyond the physical-lane items:
+
+- **No payment, wallet or fund-safety coverage** — nothing here mints, swaps
+  or melts; the cloud-lab conformance lane owns that surface.
+- **apk's scriptlet stderr is suppressed by apk-tools itself** — the gate
+  observes the guidance text only if apk chooses to surface it; its absence is
+  recorded as an observed fact, not assumed either way.
+- **x86-64 VM only** — target-specific scriptlet or kernel behavior on
+  mips/arm is not exercised by this lane.
+- **One install + one reboot** — no upgrade path, no soak, no repeated
+  install loops (the install-path lane on the PRTA bench owns upgrade/parity).
+

tests/installer-contract/checks.sh

@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# checks.sh — pure installer-contract check logic, sourced by run.sh and
+# selftest.sh. No VM access here: every function takes captured text
+# (uci show / logread / apk output) so the offline selftest can drive the
+# same code paths with planted fixtures (#899, #911 controls).
+#
+# A check function emits exactly one verdict line on stdout:
+#   <PASS|FAIL> <detail...>
+# The caller maps it onto a check id via record(). NOT_VERIFIED/SKIP are
+# lane-level statuses (environment could not provide input); they are
+# decided by run.sh, never by these functions.
+
+# ic_section_ports <uci-show-uhttpd-text> <section> — ports that section
+# listens on, space-separated. Handles multi-value uci list lines such as
+#   uhttpd.main.listen_http='0.0.0.0:8090' '[::]:8090'
+ic_section_ports() {
+    printf '%s\n' "$1" | awk -v sec="$2" '
+        $0 ~ ("^uhttpd\\." sec "\\.listen_(http|https)=") {
+            line = $0
+            sub(/^[^=]*=/, "", line)
+            n = split(line, tok, / /)
+            for (i = 1; i <= n; i++) {
+                t = tok[i]
+                gsub(/\x27/, "", t)
+                if (t ~ /:[0-9]+$/) { sub(/.*:/, "", t); print t }
+            }
+        }'
+}
+
+# ic_uhttpd_sections <uci-show-uhttpd-text>
+# The #899 invariant: no shipped uhttpd section may end the install
+# listenerless (uhttpd exits with "No sockets bound" and procd crash-loops
+# a bare instance). Port liveness alone cannot detect a torn section whose
+# ports are claimed — or dropped — by another instance; this reads the
+# config itself.
+ic_uhttpd_sections() {
+    local text="$1" listenerless="" secs sec ports detail
+    secs="$(printf '%s\n' "$text" | sed -n 's/^uhttpd\.\([^.]*\)\..*/\1/p' | sort -u)"
+    [ -n "$secs" ] || { echo "FAIL 'uci show uhttpd' produced no sections — cannot verify the #899 listenerless-section invariant"; return; }
+    for sec in $secs; do
+        ports="$(ic_section_ports "$text" "$sec" | tr '\n' ' ' | sed 's/ *$//')"
+        [ -n "$ports" ] || listenerless="$listenerless $sec"
+    done
+    if [ -n "$listenerless" ]; then
+        echo "FAIL listenerless uhttpd section(s):$listenerless — the #899 tear (an instance with no listen_http/listen_https crashes with 'No sockets bound')"
+    else
+        detail="$(for sec in $secs; do printf '%s=%s ' "$sec" "$(ic_section_ports "$text" "$sec" | sort -n | tr '\n' ',' | sed 's/,$//')"; done | sed 's/ *$//')"
+        echo "PASS every uhttpd section holds listeners ($detail)"
+    fi
+}
+
+# ic_port_ownership <mapping:board|luci> <uci-show-uhttpd-text>
+# The entry-port mapping: board mode — uhttpd.main on 8090/8443 with the
+# admin instance on the entry pair 8080/443; luci mode — the mirror image.
+ic_port_ownership() {
+    local mapping="$1" text="$2" main_ports admin_ports missing="" pair
+    case "$mapping" in
+        board) pair="main:8090 main:8443 admin:8080 admin:443" ;;
+        luci)  pair="main:8080 main:443 admin:8090 admin:8443" ;;
+        *) { echo "FAIL unknown mapping '$mapping' — ownership cannot be verified"; return; } ;;
+    esac
+    main_ports="$(ic_section_ports "$text" main | tr '\n' ' ')"
+    admin_ports="$(ic_section_ports "$text" admin | tr '\n' ' ')"
+    local sp sec port
+    for sp in $pair; do
+        sec="${sp%%:*}"; port="${sp##*:}"
+        case " $(if [ "$sec" = main ]; then printf '%s' "$main_ports"; else printf '%s' "$admin_ports"; fi) " in
+            *" $port "*) : ;;
+            *) missing="$missing $sec:$port" ;;
+        esac
+    done
+    if [ -n "$missing" ]; then
+        echo "FAIL mapping=$mapping but the expected owners are absent:$missing (main='[$main_ports]' admin='[$admin_ports]')"
+    else
+        echo "PASS mapping=$mapping: main=[$main_ports] admin=[$admin_ports]"
+    fi
+}
+
+# ic_crash_logs <logread-text>
+# Crash-loop signatures the lane has actually seen: uhttpd's zero-listener
+# exit (#899) and kernel-level process deaths.
+ic_crash_logs() {
+    local text="$1" n_sockets n_seg
+    n_sockets="$(printf '%s\n' "$text" | grep -c 'No sockets bound' || true)"
+    n_seg="$(printf '%s\n' "$text" | grep -c -E 'Segmentation fault|core dump' || true)"
+    if [ "$n_sockets" -gt 0 ] || [ "$n_seg" -gt 0 ]; then
+        echo "FAIL crash signatures in logread: 'No sockets bound'=$n_sockets, segfault/core=$n_seg"
+    else
+        echo "PASS no crash signatures ('No sockets bound'=0, segfault/core=0)"
+    fi
+}
+
+# ic_preinst_refusal <apk-exit-code> <refuse-log-text> <registered:0|1> <running:0|1>
+# The #907 invariant as #911 found it in the field: the REQUIRED outcome is
+# a clean refusal (nonzero exit, guidance visible, nothing registered). The
+# OBSERVED apk-era reality may instead be the #911 shape — apk-tools 25.12
+# executes pre-install at commit time, records a scriptlet error, suppresses
+# the scriptlet's stderr, and COMPLETES the transaction (package registered,
+# service started) while still exiting 1. That shape is reported faithfully
+# here as a FAIL with the observed facts — never massaged into a pass, and
+# never hidden behind the exit code alone.
+ic_preinst_refusal() {
+    local ec="$1" log="$2" registered="$3" running="$4"
+    local guidance=0
+    printf '%s' "$log" | grep -q "root has no password — refusing to install" && guidance=1
+    if [ "$ec" -ne 0 ] && [ "$guidance" = 1 ] && [ "$registered" = 0 ]; then
+        echo "PASS clean refusal on the empty-root device (exit $ec, guidance printed, nothing registered)"
+        return
+    fi
+    local shape
+    shape="exit=$ec guidance=$([ "$guidance" = 1 ] && echo visible || echo suppressed) registered=$([ "$registered" = 1 ] && echo yes || echo no) service=$([ "$running" = 1 ] && echo running || echo stopped)"
+    if [ "$ec" -ne 0 ] && [ "$registered" = 1 ]; then
+        echo "FAIL required refusal did not happen — apk-tools completed the transaction despite the failing preinst (#911): $shape"
+    elif [ "$ec" -eq 0 ]; then
+        echo "FAIL apk add exited 0 on an empty-root device — the preinst gate did not even fire: $shape"
+    else
+        echo "FAIL refusal incomplete: $shape"
+    fi
+}

tests/installer-contract/evidence/2026-10-13-ai-legion-rc4/iccheck.log

@@ -0,0 +1,47 @@
+ICCHECK preflight:ssh PASS 127.0.0.1:22222 reachable after serial bootstrap (fresh-boot VM)
+ICCHECK preflight:apk-era PASS OpenWrt 25.12.5 (apk era)
+ICCHECK preflight:openssl PASS lane host has openssl (credential comparison)
+ICCHECK provenance:artifact-sha256 PASS tollgate-wrt-0.6.0_rc4-r0.apk sha256 79e67a5914ed6c343d2027a407b93405d60094d708cd349e9405b7051f51e08c
+ICCHECK provenance:source-sha PASS artifact recorded as built from 6b572ddaeeac0ccede4d9a09acad94668baaa2c6
+ICCHECK pristine:factory-reset PASS firstboot + forced reboot, ssh back
+ICCHECK pristine:verified PASS no tollgate-wrt registered and root credential empty
+ICCHECK install:transfer PASS tollgate-wrt-0.6.0_rc4-r0.apk sha256 79e67a5914ed6c343d2027a407b93405d60094d708cd349e9405b7051f51e08c
+ICCHECK contract:preinst-refuses-empty-root FAIL apk add exited 0 on an empty-root device — the preinst gate did not even fire: exit=0 guidance=suppressed registered=yes service=running
+ICCHECK flow:operator-sets-password PASS root password set by the operator before install
+ICCHECK install:apk-add PASS apk add exit 0 after the operator set the credential, tollgate-wrt registered
+ICCHECK contract:package-sets-no-credential PASS root's hash is byte-identical across the install — nothing was applied, replaced or generated
+ICCHECK contract:no-provisional-marker FAIL provisional marker exists on a pristine install — a credential was generated
+ICCHECK contract:no-plaintext PASS the operator's root password never appears in the install output
+ICCHECK services:tollgate-wrt PASS running: API :2121 listening and /var/run/tollgate.sock present
+ICCHECK services:nodogsplash PASS nodogsplash running
+ICCHECK services:uhttpd PASS uhttpd running
+ICCHECK listeners:80 PASS listening
+ICCHECK listeners:443 PASS listening
+ICCHECK listeners:2050 PASS listening
+ICCHECK listeners:2051 PASS listening
+ICCHECK listeners:8080 PASS listening
+ICCHECK listeners:8090 FAIL nothing listening on :8090
+ICCHECK listeners:8443 FAIL nothing listening on :8443
+ICCHECK http:guest-entry PASS GET / on :80 answered
+ICCHECK http:admin-tls PASS admin pair answers TLS on :443
+ICCHECK mapping:entry-ui PASS board
+ICCHECK listeners:uhttpd-sections FAIL listenerless uhttpd section(s): defaults main — the #899 tear (an instance with no listen_http/listen_https crashes with 'No sockets bound')
+ICCHECK mapping:port-ownership FAIL mapping=board but the expected owners are absent: main:8090 main:8443 (main='[]' admin='[8080 8080 443 443 ]')
+ICCHECK logs:crash-signatures FAIL crash signatures in logread: 'No sockets bound'=5, segfault/core=0
+ICCHECK persistence:credential PASS root credential survived the reboot
+ICCHECK persistence:mapping PASS entry-ui mapping stable (board)
+ICCHECK persistence:services PASS tollgate-wrt auto-started after reboot
+ICCHECK persistence:listeners:80 PASS listening after reboot
+ICCHECK persistence:listeners:443 PASS listening after reboot
+ICCHECK persistence:listeners:2050 PASS listening after reboot
+ICCHECK persistence:listeners:2051 PASS listening after reboot
+ICCHECK persistence:listeners:8080 PASS listening after reboot
+ICCHECK persistence:listeners:8090 FAIL nothing listening on :8090 after reboot
+ICCHECK persistence:listeners:8443 FAIL nothing listening on :8443 after reboot
+ICCHECK persistence:uhttpd-sections FAIL listenerless uhttpd section(s): defaults main — the #899 tear (an instance with no listen_http/listen_https crashes with 'No sockets bound')
+ICCHECK persistence:port-ownership FAIL mapping=board but the expected owners are absent: main:8090 main:8443 (main='[]' admin='[8080 8080 443 443 ]')
+ICCHECK persistence:crash-signatures FAIL crash signatures in logread: 'No sockets bound'=2, segfault/core=0
+evidence: /home/ubuntu/ic-gate/evidence-run2 (admin/wifi values: root-only /home/ubuntu/ic-gate/evidence-run2/admin-password)
+ICRESULT total=43 pass=31 fail=12 skip=0 not_verified=0
+ICFAILED contract:preinst-refuses-empty-root contract:no-provisional-marker listeners:8090 listeners:8443 listeners:uhttpd-sections mapping:port-ownership logs:crash-signatures persistence:listeners:8090 persistence:listeners:8443 persistence:uhttpd-sections persistence:port-ownership persistence:crash-signatures
+ICEXIT 1