Priority: P2 — flash durability & corruption-recovery audit for wallet.db and new durable stores (G06/G09 add writes; this issue bounds the write cost and the blast radius). #505

Activity

1. Amperstrand commented on Oct 8, 2026

   

   Amperstrand

   on Oct 8, 2026

   CollaboratorAuthor

   More actions

   Flash-durability audit — every persistent write path, classified (code-verified at main @ cb4db7f)

   Scope 1 of the issue's acceptance criteria ("audit table: file, write path, fsync, atomicity, repair"). Findings below come from walking every runtime write site in src/ plus the gonuts-tollgate v0.10.0 storage layer (bbolt v1.4.0). The provably-safe fixes and the wallet check tooling land in the linked PR; measurement and the power-loss campaign need lab hardware and are staged as follow-ups at the bottom.

   Audit table

   File	Write mechanism	fsync	Atomic (old-or-new)	Torn-write consequence	Recovery
   wallet.db	bbolt Update() per money op via gonuts InitBolt; bolt.Open(path, 0600, nil) — defaults hold: NoSync=false (fdatasync per commit), NoGrowSync=false (fsync on grow), freelist synced	✅ per commit	✅ by design (double meta pages)	torn page → structurally broken DB	new: tollgate wallet check + runbook (was: nothing)
   config.json	SaveConfig: temp+fsync+rename, in-place fallback for bind-mounted inodes (#402)	✅	✅	none (old or new)	loader backup-and-defaults as last resort
   identities.json	was plain os.WriteFile — ❌ no fsync, ❌ not atomic → fixed in the PR (writeFileDurably)	was ❌ / now ✅	was ❌ / now ✅	worst in repo: file carries the Nostr signing key (owned_identities[0].privatekey, main.go:1829) and payout Lightning addresses; torn write → boot-time backup-and-factory-defaults = silent npub swap + payout target revert	corrupt copy lands in config_backups/; identity must be re-configured
   install.json	was plain os.WriteFile → fixed in the PR (same helper, bind-mount fallback keeps the cloud-lab lane working)	was ❌ / now ✅	was ❌ / now ✅	benign: self-heals to defaults, install_time restamped	automatic
   Lightning quote_store.json	temp+fsync+rename per quote transition	✅	✅	none	rewrite from mint state
   receive-intent journal (#502)	temp+fsync+rename, full-file rewrite per intent transition	✅	✅	none	pending→reconcile at restart (by design)
   owed-grant store	temp+fsync+rename, full-file rewrite	✅	✅	none	owed grants re-drive
   drain journal (#433, CLI)	O_APPEND + fsync per line; write-only in code (operator evidence artifact, no in-code reader)	✅	n/a (append)	torn final line possible	operator-side: skip malformed tail line when reading by hand
   config_backups/	os.Rename of the invalid file (backupAndLog)	❌ (dir entry)	✅ (rename)	rename lost on cut → original invalid file stays → self-heal retries next boot	automatic

   Shared residuals, accepted and documented: (a) none of the tmp+rename writers fsync the directory after rename — a lost rename means the previous version survives (old-or-new, never torn; the safe direction on flash); (b) the three merchant-side stores (quote_store, receive-intent, owed-grant) each implement the atomic pattern inline — duplication is noted, consolidation onto a shared helper is deliberate follow-up scope; (c) G06/G09 persistent session/payment stores are not landed — when they land, the tmp+fsync+rename pattern above is the entry bar, not an option.

   Findings that change assumptions

   1. bbolt v1.4 does not return corruption as errors — it panics and faults. Seeded-corruption testing (truncated file; XOR'd meta pages) produced both a recoverable panic: assertion failed at open AND an unrecoverable SIGBUS (unexpected fault address … fatal error: fault) inside Tx.Check's goroutine. Any in-process corruption checker dies on exactly the inputs it exists to judge — which is why the new tollgate wallet check runs its check in a forked worker whose death is itself a corruption verdict. This is also a finding for the daemon: a torn wallet.db can crash the merchant at startup rather than degrade politely (jffs2-adjacent robustness, worth its own issue).
   2. The issue's "verify our open flags don't disable fsync" is confirmed: gonuts passes nil options; bbolt v1.4.0 defaults are fsync-on-commit and fsync-on-grow. No action needed.
   3. wallet check proves structure, not bytes: bbolt v1.4 has no data-page checksums, so silent bit-rot inside leaf values (proofs, keyset metadata) is undetectable by tx.Check(). The command help and runbook say so explicitly; hash-compare against backups is the byte-level answer.
   4. identities.json was the audit's top silent-loss risk (signing key + payout addresses behind a non-atomic write with a factory-default self-heal) — now atomic like config.json.

   Delivered in the PR (provably safe + tested, no engine changes)

   • writeFileDurably in config_manager: one shared temp+fsync+rename helper (with the pinned-inode in-place fallback) now behind SaveConfig, SaveIdentities, SaveInstallConfig; the config silently reset to factory-default mint after daemon exit during degraded mode #402-era tests (atomicity, EBUSY fallback) pass unchanged through the shared path, plus new tests pinning the same contract for identities/install and the old-file-survives-total-failure case.
   • tollgate wallet check (src/cmd/tollgate-cli): offline, read-only bbolt consistency check in a crash-isolated worker; healthy/corrupt/missing verdicts unit-tested with seeded DBs (including both crash shapes) and verified end-to-end against the built binary; adds bbolt v1.4.0 to the CLI module (version-synced, deps-sync check green).
   • docs/wallet-recovery-runbook.md: the backup → check → verdict paths (intact restart / corrupt re-init + out-of-band fund recovery), the structure-vs-bitrot caveat, and why the config files self-heal; linked from the operator guide.

   Remaining acceptance items (need hardware — staged, not dropped)

   • Write amplification: /proc/<pid>/io write_bytes deltas over N payments/payouts/session updates per store, on the lab fleet; publish numbers and set erase-cycle budgets per device class (8 MB NOR vs SD-class).
   • Power-loss campaign: N=100 cut-during-write cycles (PRTA smart-plug or QEMU quit) across commit/grow/rename points, post-boot wallet check + balance assertion each cycle. Finding Add install scripts for basic implementation #1 above suggests the campaign will also probe daemon startup behavior on a torn DB.
   • Receive-intent/owed-grant full-file-rewrite amplification grows with journal length — quantify in the measurement pass and switch to an append/compact scheme if the numbers demand it (fund-safety rules apply; out of this PR's scope).
2. Amperstrand commented on Oct 8, 2026

   

   Amperstrand

   on Oct 8, 2026

   CollaboratorAuthor

   More actions

   Audit addendum — five write paths the first pass missed (a completeness re-sweep caught them; the table above claimed exhaustiveness it didn't have. Corrections in place, fixes in the PR):

   File	Write mechanism	fsync	Atomic	Consequence / fix
   cdk lane's mnemonic file (src/tollwallet/cdk_wallet.go, behind cdk_wallet tag)	was plain os.WriteFile of the seed phrase at first generation	was ❌	was ❌	a torn first write bricks the wallet at every later boot (load of a corrupt existing seed errors out — loud, but unrecoverable without the seed). Fixed in the PR: writeMnemonicDurably (temp+fsync+rename, 0600), unit-tested under the cdk_wallet tag
   token-recovery journal (src/upstream_session_manager/token_recovery.go)	O_APPEND line of bearer tokens from failed payments	was ❌	n/a (append)	an unsynced append is a token the operator can never recover after a power cut. Fixed in the PR: f.Sync() before close, the drain journal's precedent
   token-fingerprint salt (src/utils/token_fingerprint.go)	O_EXCL create-once, hex-encoded	was ❌	n/a (create-once)	a lost salt on first boot silently changes every fingerprint already recorded — the exact correlation break the hex encoding exists to prevent. Fixed in the PR: f.Sync() before close
   discovery probe log (src/wireless_gateway_manager/discovery_log.go:135,200)	append + size-rotate via rename (.old)	❌ (log — accepted)	✅ (rotate rename)	diagnostics only; amplification-relevant: probe logging cadence on flash belongs in the measurement pass, no code change
   (discovery_log rotation target .old covered above)

   Methodology correction: the first pass walked the named stores and the save/sync/rename call sites in the packages that own them; the re-sweep grepped the write-call inventory (os.WriteFile|Create|CreateTemp|OpenFile|Rename|Sync) across all of src/ non-test code — that difference is where these five lived (a build-tagged lane, and three packages outside the store-owning ones).
3. Amperstrand

   added 2 commits that reference this issue on Oct 8, 2026

   fix(wallet,utils): durable mnemonic + fsync'd token-recovery and salt…

   ...

   234f3b9

   changelog: fold the OpenTollGate#505 addendum fixes into the 743 frag…

   ...

   6009287