feat(facade): check repo size and enforce clone limit before cloning (#458) - #459
Open
Dipro-cyber wants to merge 2 commits into
Open
feat(facade): check repo size and enforce clone limit before cloning (#458)#459Dipro-cyber wants to merge 2 commits into
Dipro-cyber wants to merge 2 commits into
Conversation
…haoss#458) Signed-off-by: Diptesh Roy <droy88333@gmail.com>
Dipro-cyber
force-pushed
the
feat/enforce-repo-clone-size-limit-458
branch
from
August 26, 2026 07:00
9ab30b8 to
b0cb8fe
Compare
Contributor
|
im a little confused on the theory of how your safety margin system is meant to work. The original idea for that is because the size as github reports it is very likely not accurate to the actual size of the whole directory that you get after running I have other branches that are very work-in-progress for refactoring a lot of CollectOSS to use Happy to chat on slack about this too, im very curious what you find |
…ent E2E worker timeout Signed-off-by: Diptesh Roy <droy88333@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Some git repositories are exceptionally large, which can rapidly exhaust local disk space when cloned. This PR introduces a configurable maximum repository clone size limit (
max_clone_size_kb) and safety margin (clone_size_safety_margin) in CollectOSS.Before running
git clone, CollectOSS queries platform APIs (GitHub REST API or GitLab API) to obtain reported repository size stats and computesestimated_size_kb = reported_size_kb * (1 + safety_margin). If the estimated clone size exceedsmax_clone_size_kb, cloning is skipped and the repository collection status is set toFailed Clone.This PR fixes #458
Notes for Reviewers
max_clone_size_kb(default0, disabled) andclone_size_safety_margin(default0.5, 50% margin) toFacadeconfig incollectoss/application/config.pyandFacadeHelper.check_repo_size_limit()and pre-clone enforcement logic incollectoss/tasks/git/util/facade_worker/facade_worker/repofetch.py.tests/test_tasks/test_git/test_repo_size_limit.py.Signed commits
Generative AI disclosure