Skip to content

feat(facade): check repo size and enforce clone limit before cloning (#458) - #459

Open
Dipro-cyber wants to merge 2 commits into
chaoss:mainfrom
Dipro-cyber:feat/enforce-repo-clone-size-limit-458
Open

feat(facade): check repo size and enforce clone limit before cloning (#458)#459
Dipro-cyber wants to merge 2 commits into
chaoss:mainfrom
Dipro-cyber:feat/enforce-repo-clone-size-limit-458

Conversation

@Dipro-cyber

Copy link
Copy Markdown
Contributor

Description
Some git repositories are exceptionally large, which can rapidly exhaust local disk space when cloned. This PR introduces a configurable maximum repository clone size limit (max_clone_size_kb) and safety margin (clone_size_safety_margin) in CollectOSS.

Before running git clone, CollectOSS queries platform APIs (GitHub REST API or GitLab API) to obtain reported repository size stats and computes estimated_size_kb = reported_size_kb * (1 + safety_margin). If the estimated clone size exceeds max_clone_size_kb, cloning is skipped and the repository collection status is set to Failed Clone.

This PR fixes #458

Notes for Reviewers

  • Added max_clone_size_kb (default 0, disabled) and clone_size_safety_margin (default 0.5, 50% margin) to Facade config in collectoss/application/config.py and FacadeHelper.
  • Added check_repo_size_limit() and pre-clone enforcement logic in collectoss/tasks/git/util/facade_worker/facade_worker/repofetch.py.
  • Added comprehensive unit tests in tests/test_tasks/test_git/test_repo_size_limit.py.

Signed commits

  • Yes, I signed my commits.

Generative AI disclosure

  • This contribution was assisted or created by Generative AI tools.
    • What tools were used? Google Antigravity
    • How were these tools used? Code implementation, unit testing, and drafting PR description.
    • Did you review these outputs before submitting this PR? Yes

@Dipro-cyber
Dipro-cyber requested a review from MoralCode as a code owner August 26, 2026 06:27
@Dipro-cyber
Dipro-cyber force-pushed the feat/enforce-repo-clone-size-limit-458 branch from 9ab30b8 to b0cb8fe Compare August 26, 2026 07:00
@MoralCode

Copy link
Copy Markdown
Contributor

im a little confused on the theory of how your safety margin system is meant to work. The original idea for that is because the size as github reports it is very likely not accurate to the actual size of the whole directory that you get after running git clone.

I have other branches that are very work-in-progress for refactoring a lot of CollectOSS to use pygit2 rather than shelling out to a git subprocess (for speed/performance). I think if we wanted the size limit configuration we offer to users to be an actual/comparable measure to real on-disk first-clone size, maybe we should see if that library gives us a way to query the actual clone size of a repo.

Happy to chat on slack about this too, im very curious what you find

…ent E2E worker timeout

Signed-off-by: Diptesh Roy <droy88333@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check repo size and enforce a limit before cloning

2 participants