Third-party audit 2026-10-05: drift detection, real transport, file modes, CSV safety

[jehrr]

Works through the third-party audit of 2026-10-05 (taken at e756e58). Every finding was checked against a live run of the current code before it was accepted, rejected or changed.

Fixed, with the evidence

Finding	Measured	Change
A page that renders but has moved a field gives silent nulls	no check existed	payload_keys_missing: the parser checks every __NEXT_DATA__ key it reads (keys, not values — a hidden count is still a legitimate "0") and names the missing ones per account in the sidecar. 0 false positives on 20 raw captures and a live page. The canary fails on it.
engine=playwright on an HTTP run	worse than reported: with the default --transport auto the sidecar said transport: "auto", naming no transport at all	transport now records what fetched the pages (http / browser / cdp), transport_requested the flag. engine keeps naming the script (see below). Live, all three engines: http for auto, browser for --transport browser.
"complete" can be read as "the whole history"	the sidecar already said so; a row did not	has_more_spotlight / has_more_highlights on the profile row, from the page's own cursors.
(not in the audit) output files are 0600	9 of 9 files on a live run under umask 022	new files get the umask mode; an existing file keeps its mode.
(not in the audit) CSV formula injection	0 of 10,554 string cells fire today	lifted verbatim from rakuten-scraper: ' prefix in CSV only, strings only, counted in csv_cells_escaped.
Three copies of the orchestration	the shared blocks are byte-identical today	a check that fails the build the moment they are not.
Python 3.9 is past security support	—	CI's newest end moves 3.12 → 3.14; see below for why 3.9 stays.

Each new check was controlled: the fix removed, the suite red on the check named for it, run with PYTHONDONTWRITEBYTECODE=1 and a cleared cache. Six of six; one control first stayed green because the planted line fell outside the block being checked — redone inside it, red.

Not changed, and why

• engine=http. The engine is still the script that ran, and which driver the browser fallback would use matters for reproducing a run. One field for two facts loses one of them; the new transport field carries the other.
• Lazy-importing Playwright for HTTP-only use. Every engine imports its driver at module level on purpose: it is what lets CI's engine-smoke job fail when an engine cannot import, and a lazy import is exactly what made that job blind in a sibling repo. An HTTP-only user needs the pip package, not a browser download.
• Refactoring the three engines into one loop. Real, and done in one sibling (binance) where the loop is shared; here it is a rewrite of the fetch path for no behaviour change. The identity check above holds the copies together until then.
• Dropping Python 3.9. It is what macOS still ships as /usr/bin/python3 — the audit itself ran on 3.9.6 — and the code needs nothing newer. The NotOpenSSLWarning in the audit's log is that system Python's LibreSSL, not this repo.
• Human-readable category labels. The ids (public-profile-category-v3-business-group) are Snapchat's own; the labels live in its front-end bundle, change per build and are localised. A mapping written here would be a guess presented as data. The raw id is the stable join key.
• Cursor pagination, a schema version, NDJSON/webhook delivery. Product decisions rather than defects; the last two are family-wide and belong in the shared core, not in one repo.

No column was removed or renamed, and exit codes are unchanged.

🤖 Generated with Claude Code