feat: implement persistent job queue with bbolt and maintenance worker #375
Conversation
|
Thank you for opening this PR! Before a maintainer takes a look, it would be really helpful if you could walk through your changes using GitHub's review tools. Please take a moment to:
More information on how to conduct a self review: This helps make the review process smoother and gives us a clearer understanding of your thought process. Once you've added your self-review, we'll continue from our side. Thank you! |
There was a problem hiding this comment.
added required dependencies for bbolt and cron
There was a problem hiding this comment.
added required dependencies for bbolt and cron
Hell1213
left a comment
Hell1213
left a comment
There was a problem hiding this comment.
This PR solves the job persistence issue cleanly. Jobs were getting lost on backend restarts, now they're saved to disk with bbolt and restored automatically. Added a maintenance worker to clean up old logs so disk doesn't fill up. Everything is configurable and backward compatible - existing code keeps working even if the database fails. Tested live and confirmed jobs survive restarts.
|
hey @its-me-abhishek , PR is ready for Review , open to any further changes required. |
|
@Hell1213, the implementation looks great. will review it once locally first and then here. |
|
Hey @its-me-abhishek, should we keep Go 1.23 or downgrade bbolt to v1.3.7 (works with Go 1.19)? The Go 1.23 bump is because bbolt v1.4.3 requires it - the link you mentioned in the issue points to use bbolt v1.4.3. |
The current job queue was volatile and jobs would be lost on backend restarts. This implements a bbolt-based persistent queue that stores jobs to disk and restores them on startup. Also added a cron-based maintenance worker that automatically cleans up old completed and failed job logs to prevent unbounded disk usage. All existing functionality is preserved and the new features are fully configurable via environment variables. Resolves CCExtractor#367
Hell1213
force-pushed
the
feat/persistent-job-queue-maintenance-worker
branch
from
95da0ee to
de09939
Compare
|
hey @its-me-abhishek , |
|
hey @its-me-abhishek ,pls check the pr ,its ready I have implemented those changes as requested |
|
@Hell1213 there seems to be some misunderstanding, please do add those updated creds to Additionally, just checked that this PR will probably require to update the Kubernetes config, as well, in order to keep that working |
thanks , apologies for misunderstanding ,I'm on it will make those changes as told . |
Fixed job restoration to prevent duplicates and added queue environment variables to production files for Docker/Kubernetes deployments.
Hell1213
force-pushed
the
feat/persistent-job-queue-maintenance-worker
branch
from
3715a97 to
546cf7c
Compare
2 participants
The current job queue was volatile and jobs would be lost on backend restarts. This implements a bbolt-based persistent queue that stores jobs to disk and restores them on startup. Also added a cron-based maintenance worker that automatically cleans up old completed and failed job logs to prevent unbounded disk usage. All existing functionality is preserved and the new features are fully configurable via environment variables.
Description
Added persistent job queue using bbolt database to store jobs on disk. Jobs now survive backend restarts and are automatically restored on startup. Implemented maintenance worker with cron scheduler to clean up old job logs and prevent disk usage issues.
Checklist
npx prettier --write .(for formatting)gofmt -w .(for Go backend)npm test(for JS/TS testing)Terminal Screenshot
Additional Notes
New environment variables added for configuration:
CLEANUP_CRON_SCHEDULE- Schedule for maintenance worker (default: daily at midnight)CLEANUP_RETENTION_DAYS- How long to keep job logs (default: 7 days)QUEUE_DB_PATH- Database file location (default: /app/data/queue.db)The implementation gracefully falls back to in-memory queue if persistent storage fails, ensuring no breaking changes.