feat(backups): delete partial uploads on failed backups and split failure diagnostics - #174
Merged
Merged
Conversation
…lure diagnostics Adapts the useful half of upstream Dokploy#4896 to the fork's already-streamed backup pipeline (the fork streams the dump ONCE through backupCommand | rcloneCommand with pipefail, so Dokploy#4896's double-dump bug does not apply here). What was missing: - A failed streamed upload can leave a TRUNCATED object in the bucket, which then counts toward keepLatestNBackups retention and can evict a good backup. - One combined error message made "dump failed" indistinguishable from "upload failed" in the backup log. Changes to getBackupCommand (packages/server/src/utils/backups/utils.ts): - Derive a cleanup command from the SAME remote path/flags/env as the rcat upload by swapping the `rcat` verb for `deletefile` (matched anywhere in the string so the crypt env prefix from buildRcloneCommand is preserved). Run it best-effort (`|| true`) on any failure path, logging that cleanup was done. - Capture the dump's exit code and stderr via temp files (+ EXIT trap) so the log distinguishes "❌ Error: Backup failed" (dump side, takes precedence) from "❌ Error: Upload to <dest> failed" (rclone side). Keeps the fork's destinationType wording (S3/SFTP/FTP). - pipefail semantics preserved: the pipeline sits on the left of `|| UPLOAD_FAILED=1`, so set -e is ignored inside it and `echo $? > status` always records the dump code; the added plumbing never masks the pipeline's real exit status. Tests (apps/dokploy/__test__/utils/backups.test.ts): dump runs exactly once; dump streamed into rclone; both failure messages present; deletefile derived from the upload path; crypt env prefix preserved in the cleanup command. Inspired by Dokploy#4896 (adapted, not a 1:1 port — the fork already streams once and wraps rclone crypt-aware).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adapts the useful half of upstream Dokploy/dokploy#4896 to this fork's backup pipeline. This is not a 1:1 port: upstream Dokploy#4896 primarily fixed a double-dump bug that this fork does not have — the fork already streams the dump once through
backupCommand | rcloneCommandwithset -eo pipefail, and already wraps rclone crypt-aware viabuildRcloneCommand/getRclonePathAndFlags. We took the two things worth keeping:keepLatestNBackupsretention and can evict a good backup. On any failure path we now delete it best-effort.❌ Error: Backup failed(dump side, takes precedence) from❌ Error: Upload to <S3/SFTP/FTP> failed(rclone side).Changes
packages/server/src/utils/backups/utils.ts—getBackupCommand:rcloneCommand(which callers construct throughbuildRcloneCommand(rclone rcat …, envVars)), swapping only thercatverb fordeletefile. Because the swap matchesrclone rcatanywhere in the string, anyRCLONE_CRYPT_*env prefix is preserved, so crypt remotes delete the correct underlying object. The rcat target is a full file path (…/<timestamp>.sql.gz), sodeletefiletargets exactly that one object. Runs best-effort (>/dev/null 2>&1 || true) and logs that cleanup was attempted.EXITtrap (adapting upstream's pattern): the dump's stderr goes to$DUMP_ERROR_FILEand its exit code to$DUMP_STATUS_FILE. Since the pipeline sits on the left of|| UPLOAD_FAILED=1,set -eis inert inside it andecho $? > statusalways records the dump's real code — pipefail semantics are preserved and nothing masks the pipeline's exit status. Dump wording kept generic (Backup failed); upload wording keeps the fork'sdestinationType(S3/SFTP/FTP).apps/dokploy/__test__/utils/backups.test.ts— extended: dump runs exactly once; dump streamed into rclone; both failure messages present;deletefilederived from the upload path; crypt env prefix preserved in the cleanup command.Four-path trace
Let
set -eo pipefail; the pipeline is{ DUMP 2>err; echo $? >status; } | RCLONE 2>&1 >/dev/null, whose left group always exits 0, so the pipeline's status reflects rclone only (UPLOAD_FAILED).DUMP_STATUSis read from the status file.UPLOAD_FAILED=0,DUMP_STATUS=0. Neitheriffires →✅ Upload … completed/Backup done ✅. No cleanup.DUMP_STATUS≠0. First branch fires (takes precedence):❌ Error: Backup failed+ dump stderr, best-effortdeletefileof the partial,exit 1.DUMP_STATUS=0,UPLOAD_FAILED=1. First branch skipped, second fires:❌ Error: Upload to <dest> failed+ rclone stderr, best-effortdeletefile,exit 1.DUMP_STATUS≠0. First branch fires (dump precedence):❌ Error: Backup failed, cleanup,exit 1. Upload branch never reached, so the operator sees the root cause (the dump) not the downstream symptom.Testing
apps/dokploy/__test__/utils/backups.test.ts+__test__/utils/encryption.test.ts: 33 passed, 0 failed locally (vitest run --config __test__/vitest.config.ts).@dokploy/servertypecheck: clean.apps/dokploytypecheck reportsCannot find module '@dokploy/server'in two unrelated files (sso.ts,schedule.ts) — a pre-existing local build-artifact/module-resolution issue in this workspace, not introduced here (this change adds no imports and only edits a shell template string). Relying on CI for the app typecheck.