Skip to content

feat(backups): delete partial uploads on failed backups and split failure diagnostics - #174

Merged
AminDhouib merged 1 commit into
canaryfrom
feat/backup-partial-object-cleanup
Jul 22, 2026
Merged

feat(backups): delete partial uploads on failed backups and split failure diagnostics#174
AminDhouib merged 1 commit into
canaryfrom
feat/backup-partial-object-cleanup

Conversation

@AminDhouib

Copy link
Copy Markdown
Member

What & why

Adapts the useful half of upstream Dokploy/dokploy#4896 to this fork's backup pipeline. This is not a 1:1 port: upstream Dokploy#4896 primarily fixed a double-dump bug that this fork does not have — the fork already streams the dump once through backupCommand | rcloneCommand with set -eo pipefail, and already wraps rclone crypt-aware via buildRcloneCommand/getRclonePathAndFlags. We took the two things worth keeping:

  1. Partial-object cleanup. A failed streamed upload can leave a truncated object at the destination. That partial then counts toward keepLatestNBackups retention and can evict a good backup. On any failure path we now delete it best-effort.
  2. Split failure diagnostics. One combined message made a dump failure indistinguishable from an upload failure. The log now distinguishes ❌ Error: Backup failed (dump side, takes precedence) from ❌ Error: Upload to <S3/SFTP/FTP> failed (rclone side).

Changes

packages/server/src/utils/backups/utils.tsgetBackupCommand:

  • Cleanup command is derived from the same already-built rcloneCommand (which callers construct through buildRcloneCommand(rclone rcat …, envVars)), swapping only the rcat verb for deletefile. Because the swap matches rclone rcat anywhere in the string, any RCLONE_CRYPT_* env prefix is preserved, so crypt remotes delete the correct underlying object. The rcat target is a full file path (…/<timestamp>.sql.gz), so deletefile targets exactly that one object. Runs best-effort (>/dev/null 2>&1 || true) and logs that cleanup was attempted.
  • Split diagnostics via temp files + an EXIT trap (adapting upstream's pattern): the dump's stderr goes to $DUMP_ERROR_FILE and its exit code to $DUMP_STATUS_FILE. Since the pipeline sits on the left of || UPLOAD_FAILED=1, set -e is inert inside it and echo $? > status always records the dump's real code — pipefail semantics are preserved and nothing masks the pipeline's exit status. Dump wording kept generic (Backup failed); upload wording keeps the fork's destinationType (S3/SFTP/FTP).

apps/dokploy/__test__/utils/backups.test.ts — extended: dump runs exactly once; dump streamed into rclone; both failure messages present; deletefile derived from the upload path; crypt env prefix preserved in the cleanup command.

Four-path trace

Let set -eo pipefail; the pipeline is { DUMP 2>err; echo $? >status; } | RCLONE 2>&1 >/dev/null, whose left group always exits 0, so the pipeline's status reflects rclone only (UPLOAD_FAILED). DUMP_STATUS is read from the status file.

  1. Success — DUMP=0, rclone=0 → UPLOAD_FAILED=0, DUMP_STATUS=0. Neither if fires → ✅ Upload … completed / Backup done ✅. No cleanup.
  2. Dump fails — DUMP≠0, rclone typically 0 (rclone streams whatever it got and may exit 0) → DUMP_STATUS≠0. First branch fires (takes precedence): ❌ Error: Backup failed + dump stderr, best-effort deletefile of the partial, exit 1.
  3. Upload fails — DUMP=0, rclone≠0 → DUMP_STATUS=0, UPLOAD_FAILED=1. First branch skipped, second fires: ❌ Error: Upload to <dest> failed + rclone stderr, best-effort deletefile, exit 1.
  4. Both fail — DUMP≠0, rclone≠0 → DUMP_STATUS≠0. First branch fires (dump precedence): ❌ Error: Backup failed, cleanup, exit 1. Upload branch never reached, so the operator sees the root cause (the dump) not the downstream symptom.

Testing

  • apps/dokploy/__test__/utils/backups.test.ts + __test__/utils/encryption.test.ts: 33 passed, 0 failed locally (vitest run --config __test__/vitest.config.ts).
  • @dokploy/server typecheck: clean.
  • Local env caveat: apps/dokploy typecheck reports Cannot find module '@dokploy/server' in two unrelated files (sso.ts, schedule.ts) — a pre-existing local build-artifact/module-resolution issue in this workspace, not introduced here (this change adds no imports and only edits a shell template string). Relying on CI for the app typecheck.

…lure diagnostics

Adapts the useful half of upstream Dokploy#4896 to the fork's already-streamed backup
pipeline (the fork streams the dump ONCE through backupCommand | rcloneCommand
with pipefail, so Dokploy#4896's double-dump bug does not apply here).

What was missing:
- A failed streamed upload can leave a TRUNCATED object in the bucket, which
  then counts toward keepLatestNBackups retention and can evict a good backup.
- One combined error message made "dump failed" indistinguishable from
  "upload failed" in the backup log.

Changes to getBackupCommand (packages/server/src/utils/backups/utils.ts):
- Derive a cleanup command from the SAME remote path/flags/env as the rcat
  upload by swapping the `rcat` verb for `deletefile` (matched anywhere in the
  string so the crypt env prefix from buildRcloneCommand is preserved). Run it
  best-effort (`|| true`) on any failure path, logging that cleanup was done.
- Capture the dump's exit code and stderr via temp files (+ EXIT trap) so the
  log distinguishes "❌ Error: Backup failed" (dump side, takes precedence) from
  "❌ Error: Upload to <dest> failed" (rclone side). Keeps the fork's
  destinationType wording (S3/SFTP/FTP).
- pipefail semantics preserved: the pipeline sits on the left of `|| UPLOAD_FAILED=1`,
  so set -e is ignored inside it and `echo $? > status` always records the dump
  code; the added plumbing never masks the pipeline's real exit status.

Tests (apps/dokploy/__test__/utils/backups.test.ts): dump runs exactly once;
dump streamed into rclone; both failure messages present; deletefile derived
from the upload path; crypt env prefix preserved in the cleanup command.

Inspired by Dokploy#4896 (adapted, not a 1:1
port — the fork already streams once and wraps rclone crypt-aware).
@AminDhouib
AminDhouib merged commit ba5bf62 into canary Jul 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant