Skip to content

Commit 1cee583

Browse files
remove kv
Signed-off-by: Jason McCallister <jason@mccallister.dev>
1 parent c021e95 commit 1cee583

12 files changed

Lines changed: 330 additions & 955 deletions

File tree

.github/workflows/ci.yml

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,5 @@ jobs:
4242
- name: Type check
4343
run: deno task check
4444

45-
- name: Test (Deno KV)
45+
- name: Test
4646
run: deno task test
47-
48-
- name: Test (Postgres)
49-
run: deno task test:pg

CHANGELOG.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,43 @@
11
# Changelog
22

3+
## [Unreleased]
4+
5+
### Breaking
6+
7+
- Removed the Deno KV driver and the `createQueue()` factory. The queue now runs
8+
exclusively on Postgres. Construct a `PostgresDriver` and pass it to
9+
`new Queue(driver)`. KV Connect on Deno Deploy did not support `enqueue`,
10+
which made the dual driver story misleading on the platform we care about
11+
most.
12+
- `JobContext` no longer exposes `kv`. Use `ctx.locks` and `ctx.counters` from
13+
middleware.
14+
- `FailedJobDriver.store(record)` is now `store(record, envelope)`. Drivers must
15+
persist the original `QueueEnvelope` so retries reconstruct dispatch metadata
16+
(max attempts, backoff schedule, unique key/ttl, chain).
17+
18+
### Added
19+
20+
- `PostgresDriver` with `LISTEN`/`NOTIFY` wakeups, polling fallback, and
21+
`SELECT FOR UPDATE SKIP LOCKED` reservation.
22+
- `reserveTtlMs` option. Reserved jobs are deleted only after the handler
23+
returns, so a worker crash leaves the row reserved until peers can reclaim it.
24+
- `tablePrefix` option (and `QUEUE_TABLE_PREFIX` env var) for sharing a database
25+
with other applications.
26+
- Auto migration on first use, with a standalone `deno task db:migrate` task and
27+
`compose.yaml` for local development.
28+
29+
### Changed
30+
31+
- Unique dispatch is now enforced via a TTL row in the locks table rather than a
32+
row level constraint. The dedupe window now matches the requested `uniqueTtl`
33+
instead of ending when the job is reserved.
34+
- The `jobs` ready index dropped its partial `WHERE reserved_at IS NULL`
35+
predicate so the reservation query (which also reclaims expired reservations)
36+
can use it.
37+
- `PgFailedStore.retry()` rebuilds the envelope from the stored `QueueEnvelope`,
38+
preserving `maxAttempts`, `backoffSchedule`, `uniqueKey`, `uniqueTtl`, and
39+
`chain`.
40+
341
## [1.0.4] - 2026-04-19
442

543
### Other

README.md

Lines changed: 27 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,26 @@
11
# @snaapi/queue
22

3-
A job queue for Deno that runs on either Postgres (default) or Deno KV. Postgres
4-
is the recommended driver for Deno Deploy and any environment where KV Connect
5-
blocks `enqueue`. The Deno KV driver remains available with no external
6-
dependencies.
3+
A Postgres backed job queue for Deno. Designed for Deno Deploy and any
4+
environment where you need durable, multi worker job processing. Uses
5+
`LISTEN`/`NOTIFY` for low latency wakeups and `SELECT FOR UPDATE SKIP LOCKED`
6+
for safe concurrent reservation.
77

88
## Install
99

1010
```ts
11-
import { Queue } from "@snaapi/queue";
11+
import { PostgresDriver, Queue } from "@snaapi/queue";
1212
```
1313

1414
## Quick Start
1515

1616
```ts
17-
import { createQueue } from "@snaapi/queue";
17+
import { PostgresDriver, Queue } from "@snaapi/queue";
1818
import type { JobHandler } from "@snaapi/queue";
1919

20-
// Reads DATABASE_URL from the environment. Postgres is the default driver.
21-
// The schema is auto-created on first use. See "Backing Stores" for the
22-
// Deno KV alternative.
23-
const queue = await createQueue();
20+
const driver = new PostgresDriver({
21+
connectionString: Deno.env.get("DATABASE_URL")!,
22+
});
23+
const queue = new Queue(driver);
2424

2525
const sendEmail: JobHandler<{ to: string; subject: string }> = {
2626
handle(payload, ctx) {
@@ -42,27 +42,9 @@ await queue.dispatch("send-email", {
4242
Note that `queue.listen()` attaches to the backing store and must run in the
4343
same long lived process. A script that only dispatches jobs and then exits will
4444
enqueue work, but nothing will consume it until a process with `listen()` is
45-
running against the same store.
45+
running against the same database.
4646

47-
## Backing Stores
48-
49-
The queue runs on either Postgres (default for `createQueue()`) or Deno KV. The
50-
same `Queue` API works against both. Wire a driver explicitly with
51-
`new Queue(driver)` or let `createQueue()` read the environment.
52-
53-
### Postgres (default)
54-
55-
```ts
56-
import { createQueue } from "@snaapi/queue";
57-
58-
// Reads DATABASE_URL from the environment.
59-
// QUEUE_DRIVER defaults to "postgres". Set QUEUE_DRIVER=deno-kv to opt out.
60-
const queue = await createQueue();
61-
queue.register("send-email", sendEmail);
62-
queue.listen();
63-
```
64-
65-
Or wire it explicitly:
47+
## Driver
6648

6749
```ts
6850
import { PostgresDriver, Queue } from "@snaapi/queue";
@@ -71,17 +53,20 @@ const driver = new PostgresDriver({
7153
connectionString: Deno.env.get("DATABASE_URL")!,
7254
pollIntervalMs: 1000, // fallback when LISTEN/NOTIFY misses (default 1000)
7355
concurrency: 1, // in-flight jobs per listener (default 1)
56+
reserveTtlMs: 5 * 60_000, // how long a worker can hold a job before peers can reclaim it
7457
tablePrefix: "snaapi_", // see "Table prefix" below
7558
});
7659
const queue = new Queue(driver);
7760
```
7861

79-
The Postgres driver wakes up via `LISTEN`/`NOTIFY` when a new job is enqueued
80-
and falls back to a polling timer for delayed jobs. Multiple worker processes
81-
can `listen()` against the same database. `SELECT FOR UPDATE SKIP LOCKED`
82-
ensures each job is delivered to exactly one worker.
62+
The driver wakes up via `LISTEN`/`NOTIFY` when a new job is enqueued and falls
63+
back to a polling timer for delayed jobs. Multiple worker processes can
64+
`listen()` against the same database. `SELECT FOR UPDATE SKIP LOCKED` ensures
65+
each ready job is reserved by exactly one worker. Reserved jobs are deleted only
66+
after the handler returns, so a worker crash leaves the row reserved until
67+
`reserveTtlMs` elapses, at which point a peer can reclaim it.
8368

84-
#### Auto migration
69+
### Auto migration
8570

8671
The Postgres driver runs `migrate()` lazily on first use. The schema is
8772
idempotent (`CREATE TABLE IF NOT EXISTS`), so it is safe to leave on across
@@ -102,7 +87,7 @@ The standalone task is also available:
10287
DATABASE_URL=postgres://... deno task db:migrate
10388
```
10489

105-
#### Table prefix
90+
### Table prefix
10691

10792
By default the driver creates four tables: `snaapi_jobs`, `snaapi_failed_jobs`,
10893
`snaapi_locks`, `snaapi_counters`. The prefix (default `snaapi_`) is
@@ -117,49 +102,23 @@ new PostgresDriver({
117102
});
118103
```
119104

120-
#### Local development
105+
### Local development
121106

122107
A `compose.yaml` ships with the project:
123108

124109
```bash
125110
docker compose up -d postgres
126111
export DATABASE_URL=postgres://queue:queue@localhost:5444/queue
127-
deno task test:pg
112+
deno task test
128113
```
129114

130-
### Deno KV
131-
132-
```ts
133-
import { Queue } from "@snaapi/queue";
134-
const kv = await Deno.openKv();
135-
const queue = new Queue(kv);
136-
```
137-
138-
Use this when running locally, on a VPS, or anywhere a writable Deno KV file is
139-
available. Deno Deploy uses KV Connect for hosted KV, which does not support
140-
`enqueue`, so reach for the Postgres driver in that environment.
141-
142-
### Switching drivers
143-
144-
1. Stop your workers.
145-
2. Drain the queue (let in-flight jobs finish, then verify the source store is
146-
empty).
147-
3. Update `QUEUE_DRIVER` (and `DATABASE_URL` for Postgres) in your environment.
148-
Postgres is the factory default; set `QUEUE_DRIVER=deno-kv` to opt out.
149-
4. If switching to Postgres, the schema migrates automatically on first use. You
150-
can also run `deno task db:migrate` ahead of time.
151-
5. Restart workers.
152-
153-
Pending jobs do not migrate between stores. If you have queued work that must be
154-
preserved, drain it before switching.
155-
156115
### Migrating from 1.x
157116

158117
Versions before 2.0 took a `Deno.Kv` directly and exposed `ctx.kv` to handlers.
159-
The constructor still accepts a `Deno.Kv` for backward compatibility, but
160-
`ctx.kv` is no longer exposed. Custom middleware that read `ctx.kv` should use
161-
the new `ctx.locks` and `ctx.counters` primitives, which work across both
162-
backing stores.
118+
The 2.x line drops Deno KV support entirely (KV Connect on Deno Deploy does not
119+
support `enqueue`). Pass a `PostgresDriver` to the `Queue` constructor and use
120+
`ctx.locks` / `ctx.counters` from middleware instead of `ctx.kv`. Pending jobs
121+
do not migrate between stores; drain the KV queue before cutting over.
163122

164123
## Dispatch Options
165124

deno.json

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,7 @@
44
"exports": "./mod.ts",
55
"tasks": {
66
"check": "deno check mod.ts",
7-
"test": "deno test --unstable-kv -A tests/queue_test.ts",
8-
"test:pg": "deno test --unstable-kv -A tests/postgres_test.ts",
9-
"test:all": "deno test --unstable-kv -A",
7+
"test": "deno test -A tests/queue_test.ts",
108
"db:up": "docker compose up -d postgres",
119
"db:down": "docker compose down",
1210
"db:migrate": "deno run -A scripts/migrate.ts",

mod.ts

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,7 @@ export { Queue } from "./src/queue.ts";
22
export { PendingDispatch } from "./src/dispatcher.ts";
33
export { FailedJobStore } from "./src/failed.ts";
44
export { rateLimit, withoutOverlapping } from "./src/middleware.ts";
5-
export { createQueue } from "./src/factory.ts";
6-
export type { CreateQueueOptions, QueueDriverName } from "./src/factory.ts";
75

8-
export { DenoKvDriver } from "./src/drivers/deno-kv.ts";
96
export { PostgresDriver } from "./src/drivers/postgres.ts";
107
export type { PostgresDriverOptions } from "./src/drivers/postgres.ts";
118

0 commit comments

Comments
 (0)