Faster runs with a warm result cache

[ondrejmirtes]

Runs with a warm result cache spent most of their time on fixed costs that did not depend on how many files changed. Each commit takes one of them away. The first commit fixes a bug in the turbo arena that also affects released versions.

Measured on Drupal core (11,194 analysed files, 28k directories), PHP 8.5, Apple M1 Pro, turbo extension on, fastest of 3 runs. "Before" is 2.3.x at 7c94037, "after" is an earlier version of this PR:

Files reanalysed	Before	After
0 (nothing changed)	3.41 s	1.58 s
10	6.88 s	2.84 s
1,124 (10%)	10.23 s	6.21 s
2,771 (25%)	13.92 s	9.89 s
5,574 (50%)	20.96 s	16.10 s
7,204 (64%)	23.34 s	19.03 s
8,956 (80%)	26.76 s	21.14 s
11,194 (result cache cleared)	24.12 s	22.08 s
11,185 (every file edited)	50.50 s	27.40 s

The "before" column up to "result cache cleared" is from an earlier session; the machine ran about 5% slower during the "after" session (the unchanged 2.3.x build too), so those gains are if anything understated. The "every file edited" row was measured in one session.

The rework after review (one stat-signature service, one way of caching directory symbols, a batch object, ArenaCache::replace()) does not change these numbers. Before vs. after the rework, medians of 12 interleaved runs each, separate tmpDirs:

Drupal core	Before rework	After rework
nothing changed	1.568 s	1.567 s
10 files edited (one worker)	2.917 s	2.907 s
250 files edited (forked workers)	4.175 s	4.154 s

WordPress core (full analysis, result cache cleared): 20.0 s -> 19.5 s without turbo, 8.2 s -> 7.9 s with turbo (a small tree - its directory walk and symbol scan were cheap to begin with).

Commits

1. Let a saved cache entry replace the one in the turbo arena - a bug fix. Cache::load() publishes every entry it reads from disk to the arena, and the first publish of a key won. FileTypeMapper checks its name scope maps after loading them, against the hashes of the files they were created from, so after a file changed the stale map got published, was rejected, and the fresh map could never replace it. Every later lookup in every worker recreated the map: with every file of Drupal core edited, the workers created 44k maps for 11k files. New ArenaCache::replace() in the extension, used by Cache::save(): a load publishes only when the key has nothing yet, a save always wins. Callers checking their entries after loading need to do nothing for it.
2. Keep directory listings between runs - finding the files walked all 28k directories every run, about a second, and the worker building the symbol index walked them again. A directory is read again only when its mtime, ctime or inode changed. Output identical to Symfony Finder, including order; Finder is still used on Windows and for stream wrappers.
3. Reuse the file hashes of the result cache while the files are unchanged - sha256 of every analysed file (0.84 s) is skipped while size, mtime, ctime, inode and device match, like git's index.
4. Store the dependency graph of the result cache with a table of paths - 430k dependency edges, each a path relativized/absolutized on every save/restore (0.5 s). Cache 194 MB -> 161 MB.
5. Leave the exported nodes in the result cache file until they are needed - 135 MB of exported nodes were decoded on restore and serialized on save; now only the changed files are decoded and the rest is copied as bytes. Peak memory on restore 380 MB -> 126 MB.
6. Do not parse a changed file when the outcome cannot add anything - when a changed file's dependents all changed too, comparing its exported nodes cannot add anything. With every file edited, restore spent 9 s parsing serially in the main process.
7. Check stat signatures in one place - FileStatSignatures::begin() takes the time before anything is read, and the reader it returns gives a signature only when it can be trusted, null otherwise (Windows, stream wrappers, a failed stat, a file modified in the second the reading began). DirectoryWalker and ResultCacheManager just compare what they get.
8. Keep the symbols found in directories between runs, checked by stat - OptimizedDirectorySourceLocatorFactory cached the symbols and checked the cache by hashing every file, except with turbo and forked workers, where every file was scanned natively on every run instead (0.7 s on Drupal) because hashing cost about as much. Now there is one way: the cache entry keeps each file's stat signature next to its hash, and the hash is only computed when the signature cannot vouch for the file. The arena records for the file hashes and symbol maps of workers that are not forked are gone. Files declaring no symbols are cached too; they used to be scanned on every run.
9. Scan the directory locators of the lazy source locator in one batch - with a single worker, each directory was scanned on its own; now they are batched like PreForkDirectorySymbolScanner does. createBatch() returns an object with createByDirectory(), createByFiles() and scan(), replacing beginBatchedScan()/flushBatchedScan().
10. Bump expected turbo version - for commit 1.

Commits 3 -> 4 -> 5 -> 6 touch the same code in ResultCacheManager and depend on each other textually. 4 and 5 each bump CACHE_VERSION; if both are kept, one bump is enough. Commit 7 builds on 2 and 3, commit 8 on 7, commit 9 on 8.

Safety

• The stat-based caches (listings, file hashes, symbols) trust a signature of size, mtime, ctime, inode and device, checked in one place (commit 7). A file or directory modified in the second the reading began is not trusted, because timestamps have a one-second resolution. On Windows nothing is trusted, since ctime there is the creation time; the hashes decide as before.
• Tried to break them with edits within one second: a fixture project edited right as a second begins, a run started on the cached tmpDir, a second edit at a random moment during that run, then the next cached run compared with a run on an empty tmpDir. The edits keep file sizes (in place), add, remove and rename files, set the mtime back to 2020, or replace a file by renaming a new one over it with the old mtime. 290 iterations with this PR, one worker and four forked workers: no stale result. The same harness against a build with the same-second check removed: 12 stale results in 55 iterations.
• An older PHPStan reading the new cache file falls back to a full analysis (tested by downgrading and upgrading on Drupal). An empty dependencies section is still written because older versions absolutize it before their cacheVersion check.
• Incremental results were compared with a full analysis on Drupal for every edit set above (10 files up to every file edited): identical errors, and the same number of reanalysed files as before these changes.
• Workers spawned instead of forked with turbo on (Windows, or OPcache on) no longer share the symbol maps and directory file hashes through the arena; each builds its maps from the cache entry, which is still shared through the arena by Cache::load().

Tests

• PHPUnit: all green. New tests for DirectoryWalker (Finder parity including dot files, VCS directories, symlinks, broken symlinks; added/removed/renamed files; replaced directories), FileStatSignatureReader, OptimizedDirectorySourceLocatorFactory (hash fallback, trusted signature, changed file, batches sharing a cache key), CachedExportedNodes, and ArenaCache::replace() in arena-smoke.php (including racing replaces).
• PHPStan self-analysis and phpcs: clean.
• Result cache E2E suite, locally on macOS: all pass except bug-14718 (GNU sed -i) and result-cache-ci-notification (needs CI env vars), both pass with those adjusted.
• E2E suite: the 5 failures locally are environmental (PHP 8.5 vs composer.json constraints, timeout and sudo missing on macOS).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdwtgiEHwkF5BTz73jgHQQ

[phpstan-bot]

This pull request has been marked as ready for review.