Skip to content

Jobs: no boot-time reconciliation — a job whose worker dies with the process stays IN_PROGRESS forever #2635

Closed

Description

@cb1kenobi

Pre-existing gap in the jobs subsystem; a prerequisite for #2632 (item 3; design note §7.5 in docs/proposals/archive-restore.md on branch design/archive-restore).

Problem

server/jobs/ has no boot-time sweep. Status transitions happen only in jobRunner (IN_PROGRESS, and ERROR on a throw — server/jobs/jobRunner.ts:108-131) and in the job worker's own finally (server/jobs/jobProcess.ts:71-88). A job whose worker dies with the process — crash, restart, or HARPER_EXIT_ON_RESTART under an orchestrator — stays IN_PROGRESS forever. create_backup and restore_backup already leave such rows today, and get_job reports them as running indefinitely.

The restart route for restore_backup (item 6) needs to reconcile its own job row at boot; without a generic sweep that would be one-off machinery for one operation.

Fix

  • Record process-instance ownership on the job row when it starts: pid plus a per-boot instance id (the componentPreparationLock pattern, so pid reuse cannot claim a stale row).
  • At boot, mark IN_PROGRESS rows whose owner is gone as ERROR (or a new INTERRUPTED status), with a message naming the interruption.
  • Handle rows that exist today with no owner recorded (treat as interrupted when their start predates this boot).
  • Defer to a live restore intent: a job named by a pending restore intent (item 6) is owned by the boot hook, not interrupted; the sweep must consult that before touching the row.

Acceptance

  • Unit: sweep with and without ownership metadata; a row named by a live intent is left alone; pid reuse with a different instance id is treated as gone.
  • Integration: kill Harper mid-create_backup, restart, get_job shows the job as interrupted rather than in progress.

Activity

  1. self-assigned this
    on Sep 16, 2026
  2. added
    bugSomething isn't working
    area:operations-apiOperations API: handlers, validation, response shape
    on Sep 16, 2026
  3. added theissue type on Sep 16, 2026
  4. added 3 commits that reference this issue on Sep 22, 2026
    502bedd
    1a92555
    de50941
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:operations-apiOperations API: handlers, validation, response shapebugSomething isn't working

Type

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions