Skip to content

perf(scheduler): publish an update only when addTask changes the first task or it is due - #67

Merged
Upd4ting merged 1 commit into
mainfrom
perf/publish-on-new-head
Nov 10, 2026
Merged

Upd4ting merged 1 commit into
mainfrom
perf/publish-on-new-head

Conversation

@Upd4ting

@Upd4ting Upd4ting commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Why

Every addTask published update on SchedulerUtil, and every listener process answered each message with an updateTimer round trip (ZRANGE). With L listeners, N adds cost N × L extra Redis round trips and listener CPU. Most of those messages changed nothing, because the new task was due after a first task that already had a timer leading to it.

What changes

  • addTask runs one Lua script, AddTaskScript: ZADD, then a read of the first member and its score. It publishes update only when the new member is now first, or the first member is already past due. The !locked rule is unchanged.
  • The past-due branch keeps main's recovery path. After a batch that rejected, or a lost message, the next addTask still wakes the listeners.
  • No export, Redis key, channel or member format changes. Old and new instances share a queue.
  • The TLA+ model gets PublishAfterWrite in AddTaskZadd and RescheduleZadd. A tie on score may publish or not. The model's index.ts:NN references are moved to the new lines.
  • Docs and skills/redis-scheduler-interface/SKILL.md describe when addTask publishes.
flowchart TD
  A["addTask"] --> S["AddTaskScript: ZADD, read first member and score"]
  S -->|"new member is first, or first is past due"| P{"locked?"}
  S -->|"otherwise"| N["no message: the first task's timer or batch ends in updateTimer, which finds the new task"]
  P -->|no| U["PUBLISH update: each listener runs updateTimer"]
  P -->|yes| N2["no message: this process's batch ends in updateTimer"]
Loading

Measurement

Workload: one producer calls addTask 10,000 times, 100 at a time, against a local Redis 7 container. Each listener is a separate Node process that called enableListener. Tasks are due an hour out, so no handler runs. The realistic shape is Date.now() + delay, so each new task is due after the queue head. Redis commands and CPU come from INFO all, and process CPU from process.cpuUsage(). There were 7 interleaved runs per side on a shared server, load average 3 to 5. Results are medians, with the range in brackets. Every run counted 10,000 queued members at the end, so there were 0 errors.

Evidence level: confirmed through benchmarks.

10k adds, 4 listeners main this PR
Listener CPU, total 1010 ms [934–1166] 3.9 ms [3.7–4.1]
Redis CPU 225 ms [201–281] 101 ms [94–107]
Producer wall time 252 ms [223–468] 115 ms [102–123]
Redis commands 10k ZADD, 10k PUBLISH, 40k ZRANGE 10k EVAL, 1 PUBLISH, 4 listener ZRANGE

The other orders, 4 listeners each:

  • Random due times: listener CPU fell from 1033 ms to 15.7 ms, and 9 messages were sent instead of 10,000.
  • Descending due times, where every add becomes the new first task, is the worst case. Listener CPU was 1022 ms on main and 1024 ms here. Redis CPU was 221 ms on main [213–240] and 242 ms here [224–251]. The script costs about 2 µs more per add when the message cannot be skipped.
  • With no listener, Redis CPU fell from 137 ms to 108 ms, because the separate PUBLISH is gone.
  • Busy queue: the same 4 listeners run every task, and each task is due 20 ms after it is added, at about 8,000 adds/s. The first task is often past due for a few ms here, so the past-due branch publishes about 2,400 times instead of 10,000. Listener CPU fell from 1,363 ms [1,330–1,410] to 698 ms [639–757], 5 interleaved runs per side, and all 10,000 tasks ran on both sides. Review measured the same trend at 1,000 adds/s with due times spread at random: with a 10 s spread, PUBLISH fell from 3,000 to 209; with 1 s, from 3,000 to 1,200; with 100 ms, from 3,000 to 1,313. The saving shrinks as the delay nears the take latency, and it is largest for tasks due well ahead of the first one.

Serial addTask latency did not change: p50 57–66 µs on both sides over 3,000 calls.

Limiter. On main, listeners spend their CPU parsing one ZRANGE reply per message per process. With this PR, adds are bound by the producer's round trip to Redis, and listeners sit idle.

Cumulative impact, estimated: about 24 µs of listener CPU saved per add per listener, plus L + 1 Redis commands. At 1,000 adds/s with 4 listeners that is about 0.1 CPU-s/s across the listeners, and 5,000 fewer Redis commands/s.

Proof

New tests in src/tests/index.test.ts, run with pnpm run build && pnpm exec ajs module test . -f dist/tests/index.test.js:

  • should publish an update only when the task becomes the first in the queue fails on main's addTask, which publishes every time:
  20 passing (955ms)
  1 failing

  1) Redis Scheduler - addTask
       should publish an update only when the task becomes the first in the queue:

      AssertionError: expected [ 'update', 'update', 'update' ] to deeply equal [ 'update', 'update' ]
  • should publish an update for a later task while the first task is overdue fails on a version that publishes only for a new first member. That was this branch's first draft, and review caught that it strands a due task after a rejected batch.

After, at this head:

    ✔ should publish an update only when the task becomes the first in the queue (56ms)
    ✔ should publish an update for a later task while the first task is overdue
  21 passing (961ms)

Formal check: node .agents/skills/formal-scheduler-delivery/check.mjs, with and without --full, exits 0. Every rule holds in every configuration, and the record matches. On scratch copies of the model:

  • PublishAfterWrite always {FALSE} gives BROKEN NoLostTask in 8 states (CI) and 11 (full).
  • Swapping its first-member answers gives the same.
  • Ties never publishing still exits 0.

pnpm run checks prints Checks: 11 passed, 0 failed.

Attention

  • A lost update message heals later than on main in one case. Suppose the first task is not yet due and its message is lost. Main healed at the next addTask of any kind. This PR heals at the next addTask after that task falls due, or when a listener's timer next fires. The reviewer's probe gave H@412 on main and H@3004 here, when the only later add came before H was due. Pub/sub was already at most once on main.
  • The past-due check uses the producer's clock. A producer ahead of the listeners by Δ publishes for a first task within Δ of falling due, and one behind delays the rescue by Δ. At NTP-level skew this is a few ms.
  • Producers now run EVAL. Listeners already did, through TakeDueTasksScript. An ACL or proxy that blocks scripts for producer-only processes would break addTask.
  • The model has no failing Redis commands, so it does not check the past-due branch. The second unit test covers it.

@Upd4ting
Upd4ting merged commit e05ff0d into main Nov 10, 2026
10 checks passed
@Upd4ting
Upd4ting deleted the perf/publish-on-new-head branch November 10, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant