fix: read nested .gitignore files in repo watcher to prevent inotify OOM (APP-4657)#12191
Draft
warp-dev-github-integration[bot] wants to merge 1 commit into
Draft
fix: read nested .gitignore files in repo watcher to prevent inotify OOM (APP-4657)#12191warp-dev-github-integration[bot] wants to merge 1 commit into
warp-dev-github-integration[bot] wants to merge 1 commit into
Conversation
APP-4657) The recursive inotify watcher in the repo_metadata crate previously only consulted the repo's root-level .gitignore when deciding which directories to descend into. In large monorepos (e.g. Figma's), node_modules and other build-artifact directories excluded only by per-package .gitignore files were recursively watched, causing the inotify EventLoop's internal HashMap<PathBuf, ...> to grow to gigabytes (4.67 GB in the reported event). Introduce repo_watch_filter_nested() that wraps the existing root-gitignore check with a lazy, per-registration cache of nested .gitignore files. When the root gitignore does not prune a directory, the filter reads and caches .gitignore files in every ancestor directory between the repo root and the path being evaluated. If any nested gitignore ignores the directory the filter prunes it, preventing inotify watches from being registered. Update both call sites (LocalRepoMetadataModel::add_repository_internal and DirectoryWatcher::start_watching_directory) to use the new filter. The original repo_watch_filter() is preserved for other callers. Heap profile evidence: - 96.64% (4.67 GB) from notify::inotify::EventLoop::add_watch/add_single_watch - 80% from Vec<u8>::clone (PathBuf cloning during HashMap rehash) - 16.5% from HashMap reserve_rehash (watch descriptor map growing unboundedly) - Root repo: /home/ubuntu/figma/figma (Figma monorepo, docker env, Linux) Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes the inotify recursive watch registration consuming gigabytes of heap on Linux when watching large monorepos (e.g. Figma's). Addresses APP-4657.
Root cause: The repo watch filter used by
LocalRepoMetadataModelandDirectoryWatcheronly consulted the repository's root-level.gitignorewhen deciding whether to descend into subdirectories. In monorepos wherenode_modulesor other large build-artifact directories are excluded only by per-package.gitignorefiles (not the root one), the inotify backend walked and registered watches for every single subdirectory, causing the internalHashMap<PathBuf, (WatchDescriptor, ...)>to grow to 4.67 GB (96.64% of the reported 4.83 GB heap).Fix: Introduce
repo_watch_filter_nested()that wraps the existing root-gitignore check with a lazy, per-registration cache of nested.gitignorefiles:.gitignorefiles in every ancestor directory between the repo root and the path being evaluated (lazily, with caching)Both call sites (
LocalRepoMetadataModel::add_repository_internalandDirectoryWatcher::start_watching_directory) are updated to use the new filter. The originalrepo_watch_filter()is preserved for other callers.Linked Issue
ready-to-specorready-to-implement.Fixes APP-4657
Testing
repo_metadatatests passcargo clippy -p repo_metadatapasses with no warningsshould_watch_prunes_gitignored_directoryand related test suiteAgent Mode
CHANGELOG-BUG-FIX: Fix excessive inotify watches on Linux when monitoring large monorepos with per-package
.gitignorefiles (e.g.node_modulesexcluded per-package, not at root)Conversation: https://staging.warp.dev/conversation/91e2c3e5-4df3-4f07-b91e-f70c77d9d4bc
Run: https://oz.staging.warp.dev/runs/019e91d1-787a-716c-abaa-f17f79a67c37
This PR was generated with Oz.