fix(backups): scope retention to the files of its own database - #4917
Open
joaquinleondev wants to merge 1 commit into
Open
fix(backups): scope retention to the files of its own database#4917joaquinleondev wants to merge 1 commit into
joaquinleondev wants to merge 1 commit into
Conversation
The object key of a backup is <service-appName>/<prefix>/<timestamp>.gz, so two backups of two databases hosted by the same service land in the same folder under names that carry no identity. keepLatestNBackups then lists that whole folder and deletes everything past its own keepLatestCount, which means the smallest keepLatestCount among the backups sharing the folder silently governs all of them: a daily backup set to keep 7 ends up with 1 file because a weekly backup of a second database in the same service keeps 2. Backup files are now named <database>-<timestamp>.gz and retention only lists that database's files, so a backup can no longer delete another one's dumps. Web server backups keep their pattern: they already live in a folder of their own, keyed by the backup's appName. Note for existing installations: files written before this change do not match the new pattern and are no longer pruned automatically; they have to be removed once, by hand. Closes Dokploy#4914
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4914
Problem
A backup's object key is
<service-appName>/<prefix>/<timestamp>.sql.gz.appNameidentifies the service, not the database, and the file name is only a timestamp — so two backups of two databases hosted by the same Postgres/MySQL/Mongo service land in the same folder under names that carry no identity.keepLatestNBackupsthen lists the whole folder and deletes everything past its ownkeepLatestCount:The list is not scoped to the backup that owns it, so the smallest
keepLatestCountamong the backups sharing the folder silently governs all of them. In my case a daily backup of a 9.4 GB database with Keep latest = 7 ended up holding a single file, because a second database in the same service had a weekly backup with Keep latest = 2. Both backups reported ✅ and the UI kept showing "7".What changed
<database>-<timestamp>.sql.gz(.bson.gzfor mongo), via a newgetBackupFileNamehelper. This mirrors what web server backups already do (webserver-backup-<timestamp>.zip) and, as a bonus, makes a shared folder readable by a human.--includeis built from the samegetBackupFilePrefix, so it only ever lists the files of the database it runs for.getKeepLatestNBackupsCommand(backup, destination)so it can be tested without hitting the database or S3;keepLatestNBackupsis otherwise unchanged.*.zippattern: their folder is keyed by the backup's ownappName, so they cannot collide with anything.Upgrade note, worth calling out: files written before this change are named
<timestamp>.sql.gz, do not match the new pattern, and therefore stop being pruned automatically — they have to be deleted once, by hand. Any fix that makes retention distinguish its own files has this property; the alternative (keeping the old pattern in the filter) would preserve the bug. Happy to add a follow-up that also prunes legacy files when a folder holds a single backup, if you think it's worth the extra query.Testing
Added
apps/dokploy/__test__/backups/retention-scope.test.ts. It runs the command produced bygetKeepLatestNBackupsCommandfor real, with anrclonestub onPATHthat honours--include(rclone's{a,b}alternation translated to bash extglob) and records whatrclone deletewas asked to remove. Given a folder holding threeapp-*dumps and twoanalytics-*dumps:appwith keep=2 deletes only the oldestapp-*file;analyticswith keep=1 deletes only the oldestanalytics-*file and never touchesapp-*;keepLatestCount;*.zip;[a-zA-Z0-9._-]becomes_).I also checked the tests fail against the current behaviour: restoring the old
--include "*.{sql.gz,bson.gz}"pattern makes exactly the two cross-database cases fail.Full suite, before and after, on the same machine (the failures are pre-existing and environmental — they need a database and a swarm manager, neither of which exists in my sandbox):
canarybiome checkclean on the touched files,tsc --noEmitclean inpackages/server.