feat(migrate): let migrations opt out of the transaction wrapper (-- taskq:no-transaction)

CONTRIBUTING.md

@@ -196,6 +196,113 @@ uv run ruff format --check .
    uv run ruff format --check .
    ```
 
+## Authoring Database Migrations
+
+TaskQ's schema evolves through SQL migration files bundled with the package and applied by the runner in `src/taskq/migrate.py`. This section is for TaskQ developers adding or changing those files. End users never author migrations — they only apply them — so operator-facing guidance (upgrading, failure recovery) lives in [docs/guides/upgrading.md](docs/guides/upgrading.md) and is out of scope here.
+
+### File Naming and Discovery
+
+Migration files live in `src/taskq/migrations/` and follow one naming convention:
+
+```
+{ver}_{nn}_{pre|post}_{description}.sql     # e.g. 01.00.03_01_pre_idempotency_scope.sql
+```
+
+- `{ver}` is three two-digit groups joined by dots (`01.00.03`), `{nn}` is a two-digit sequence, and `{description}` is lowercase `[a-z0-9_]+`. A filename that does not match this pattern fails discovery with a `ValueError`.
+- `discover()` finds every `*.sql` file in the package automatically — there is no registry or index to update — and returns them sorted by version, with `pre` before `post` at the same version. The zero-padding is what makes the lexicographic sort chronological, so keep the two-digit groups.
+- Migration versions are numbered independently of the package version: releases v0.1.0–v0.2.2 all shipped the same two migrations (`01.00.00_01` and `01.00.01_01`). Pick the next free `{ver}_{nn}`; do not try to mirror the package's semver.
+
+### The Default Transaction Wrapper
+
+By default each migration file runs inside its own transaction, so a failure rolls the whole file back. Keep migrations transactional unless Postgres makes that impossible.
+
+### Non-Transactional Migrations (`-- taskq:no-transaction`)
+
+Postgres forbids some statements inside a transaction block — `CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, several `ALTER TYPE`/`VACUUM` forms. A migration that needs one of these opts out of the wrapper with a header directive:
+
+```sql
+-- taskq:no-transaction
+```
+
+The usual motivation is lock duration: a plain `CREATE INDEX` holds a lock that blocks readers and writers for the whole build, which is not acceptable on hot tables such as `jobs` or `job_events`. The `CONCURRENTLY` forms avoid that lock but must run outside a transaction.
+
+Directive parsing rules (implemented by `_uses_transaction` in `src/taskq/migrate.py`):
+
+- **Leading comment block only.** Only blank lines and `--` line comments before the first SQL token are scanned; a directive later in the file is ignored, so a stray mention cannot silently flip a migration's semantics.
+- **Prefix match.** A trailing note after the token is fine and encouraged: `-- taskq:no-transaction — CIC cannot run inside a transaction`.
+- **Case-insensitive**, like SQL itself.
+- **Token-bounded.** A `\b` word boundary stops prefix drift, so `-- taskq:no-transactional` does NOT opt out.
+- **Typos surface.** A `taskq:` header line that doesn't match the directive (a typo, or a lookalike such as `no-transactional`) logs a `migration-directive-unrecognized` warning naming the file, instead of silently running the migration transactional — the worst outcome for a lock-duration-motivated migration.
+- Files are decoded as `utf-8-sig`, so a BOM from a Windows editor cannot silently disable the directive.
+
+A non-transactional file is executed statement by statement (a multi-statement string would run as one implicit transaction and defeat the opt-out). Two rules keep that safe:
+
+1. **The migration must be idempotent and re-runnable.** Nothing rolls back: a mid-file failure leaves earlier statements in place, the ledger records the migration only after every statement succeeds, and the migration is re-executed on the next `migrate up`. Write every statement to tolerate being re-run (`IF NOT EXISTS`, guarded inserts, ...).
+2. **Drop interrupted-build debris first.** An interrupted `CREATE INDEX CONCURRENTLY` leaves an INVALID index behind, and `IF NOT EXISTS` alone would then silently skip rebuilding it. Pair the statements:
+
+   ```sql
+   -- taskq:no-transaction
+   -- NOT redundant with IF NOT EXISTS below: an interrupted CREATE INDEX
+   -- CONCURRENTLY leaves an INVALID index that IF NOT EXISTS alone would
+   -- silently skip rebuilding, so drop the debris first.
+   DROP INDEX CONCURRENTLY IF EXISTS "{schema}".jobs_foo_idx;
+   CREATE INDEX CONCURRENTLY IF NOT EXISTS jobs_foo_idx ON "{schema}".jobs (foo);
+   ```
+
+Transaction-control statements are rejected in non-transactional files: `BEGIN`, `COMMIT`, `ROLLBACK` (plus their Postgres synonyms `END`, `ABORT`, `START`), `SAVEPOINT`, `RELEASE`, and `SET LOCAL` / `SET TRANSACTION`. The transaction-control group would silently re-open a transaction — defeating `CONCURRENTLY` and, on failure, poisoning the caller's connection. `SET LOCAL`/`SET TRANSACTION` are rejected for the opposite reason: outside a transaction they are silent no-ops, so you would believe e.g. `statement_timeout` was disabled for a long build when it was not. The guard rejects the whole file before any statement executes. Plain `SET` / `SET SESSION` and `CHECKPOINT` are deliberately allowed: they are session-scoped (or transaction-agnostic) and behave identically either way.
+
+### The `{schema}` Placeholder
+
+Always qualify objects with the literal `"{schema}"` token — the runner substitutes the configured schema name at apply time (after validating it as an identifier), which is what lets multiple TaskQ instances share one Postgres cluster. Never hardcode a schema name. Substitution uses `str.format`, so a literal curly brace in the SQL is written doubled (`{{` / `}}`).
+
+### Pre and Post Phases for Rolling Deploys
+
+Destructive changes are split across a deploy boundary:
+
+- **`pre`** adds structures that BOTH old and new code tolerate (e.g. a new index alongside the old one). Safe to apply before or during the code rollout.
+- **`post`** removes the old structures after every worker in the fleet has been upgraded.
+
+The runner enforces the ordering: a post-phase migration is refused until its same-version pre-phase counterpart is applied (or applies earlier in the same run). See `01.00.03_01_pre_idempotency_scope.sql` for the canonical example, including the "PHASE OBLIGATIONS" header that spells out the deployment sequence.
+
+### Never Edit a Released Migration
+
+The ledger (`{schema}.schema_migrations`) records each applied migration under its `{ver}_{nn}:{phase}` key together with a SHA-256 checksum of the rendered SQL. Editing a file after it shipped makes the checksum drift, and the runner logs a `migration-checksum-drift` warning on every subsequent apply. Treat released migrations as frozen — releases v0.1.0 through v0.2.2 shipped only `01.00.00_01` and `01.00.01_01`, and those files have not changed since. Fix forward with a new migration instead.
+
+### File Header Convention
+
+Every migration file opens with a `--` header comment block, modeled on the existing files:
+
+1. One short paragraph stating what the migration does and why.
+2. The forward-only reminder: "Forward-only; there is no down migration. To revert, restore from backup."
+3. The substitution note: 'The literal "{schema}" token is substituted at apply time by the migration runner.'
+
+Migrations with operational impact add named ops-note sections under banner comments — see the `-- ── Maintenance-window caveat ... ──` block in `01.00.02_01_pre_job_events_outbox.sql`. State the lock impact plainly: which lock the statement takes, on which table, how long it can be held (build time is proportional to row count), what it blocks (readers/writers, hot paths), and what operators of large deployments should do instead (e.g. build the index manually with `CONCURRENTLY` during a maintenance window). Phase-coupled migrations additionally document the deployment sequence and the failure modes of applying out of order (see "PHASE OBLIGATIONS" in `01.00.03_01_pre_idempotency_scope.sql`).
+
+### What CI Proves for You
+
+- **tests/test_migrations.py** applies the full bundled set to a real PostgreSQL and pins that a second `apply_pending` is a no-op.
+- **tests/test_migrations_populated.py** applies every bundled migration one step at a time onto seeded data. New migrations join this harness automatically — unknown keys get the generic per-step invariants — and its seeder intersects live columns from `information_schema`, so a future NOT NULL-without-default column fails there as the alarm. Review `_MIGRATION_SPECIFIC_CHECKS` when adding a migration whose populated-DB effect deserves a sharper assertion.
+- **`test_bundled_migrations_are_all_transactional`** (tests/test_migrations_unit.py) pins that no bundled migration carries the no-transaction directive yet. It becomes an allowlist once PRs #25/#27 land the first bundled no-transaction migration.
+- **Directive-parsing, transaction-control-guard, and statement-splitter unit tests** (tests/test_migrations_unit.py) pin every rule quoted above.
+- **`test_discover_directive_parsing_applies_end_to_end`** (tests/test_migrate_no_transaction.py) exercises the real `discover()` parse against a real database, directive-with-trailing-note included.
+- **`test_migrate_up_cli_reports_failed_no_transaction_migration`** (tests/test_migrate_no_transaction.py) proves the `migrate up` failure report end to end: what failed, what state it left the schema in, and the one action to take.
+- **`test_interrupted_concurrent_build_remedy_drop_and_rebuild`** (tests/test_migrate_no_transaction.py) stages a real interrupted `CREATE INDEX CONCURRENTLY` and proves the drop-and-rebuild remedy replaces the INVALID index with a valid one.
+
+### Local Loop
+
+```bash
+uv run pytest tests/test_migrations.py tests/test_migrations_populated.py tests/test_migrations_unit.py tests/test_migrate_no_transaction.py -q
+```
+
+The integration files need Docker, like the rest of the integration suite.
+
+### Non-Goals
+
+- **No downgrade machinery.** There is no `down` operation and none is planned; reverting means restoring from a database backup.
+- **No auto-retry in the runner.** Re-running `migrate up` (or restarting the worker) IS the heal — migrations are idempotent by contract, so a failed apply is fixed forward, not retried by the framework.
+- **No `migrate status` health probing.** `migrate status` lists applied and pending migrations; it deliberately does not validate or repair schema state.
+- **End users never author migrations.** This section covers the bundled files only; there is no user-defined migration hook.
+
 ## Pull Request Process
 
 ### Before Submitting

docs/architecture.md

@@ -391,12 +391,18 @@ Crashes are rare so the added wakeup cost is low, but the channel's meaning
 has broadened.
 
 The index added by migration `01.00.02_01_pre_job_events_outbox.sql` uses
-`CREATE INDEX` (not `CONCURRENTLY`, since `migrate.py` applies each migration
-inside a transaction and Postgres forbids `CONCURRENTLY` there) and takes an
-exclusive lock on `job_events` for the duration of the build, which can stall
-writes to that heavily-written table — operators with a large/populated table
-should run the equivalent `CREATE INDEX CONCURRENTLY` manually during a
-maintenance window (see the migration file for details).
+`CREATE INDEX` (not `CONCURRENTLY`) and takes an exclusive lock on
+`job_events` for the duration of the build, which can stall writes to that
+heavily-written table. The migration runner supports a per-migration opt-out
+from its default transaction wrapper — the `-- taskq:no-transaction` header
+directive, which unlocks `CONCURRENTLY` forms (see
+[Non-transactional migrations](guides/upgrading.md#non-transactional-migrations)) —
+but bundled migrations deliberately remain transactional (pinned by
+`test_bundled_migrations_are_all_transactional`), so operators with a
+large/populated table should still run the equivalent `CREATE INDEX
+CONCURRENTLY` manually during a maintenance window (see the migration file
+for details). Future index migrations on hot tables can adopt the directive
+instead.
 
 ---
 
@@ -961,8 +967,10 @@ These invariants must remain true across all changes.
    documentation; the `_single_threaded()` guard is a no-op.
 
 7. **Migration files are append-only** — never modify an applied migration.
-   The migration runner stores a checksum of each applied file in
-   `schema_migrations` and rejects re-runs with a checksum mismatch.
+   The migration runner stores a SHA-256 checksum of each applied file's
+   rendered SQL in `schema_migrations` and logs a `migration-checksum-drift`
+   warning when an applied file no longer matches, so tampering surfaces in
+   logs (drift is warned on, not rejected — applied migrations never re-run).
 
 8. **`BACKEND_PROTOCOL_VERSION` is checked at import time** — both
    `PostgresBackend` and `InMemoryBackend` assert the version constant at module

docs/guides/cli.md

@@ -148,6 +148,8 @@ taskq migrate up [OPTIONS]
 
 The command is idempotent: each migration is recorded in `{schema}.schema_migrations` and is skipped on subsequent runs. Running `taskq migrate up` with no options applies all pending migrations.
 
+On failure (exit code 1) the command never prints a traceback. When a migration fails during apply, the command diagnoses itself: it names the failed migration, reports the state the schema was left in — a clean rollback for transactional migrations, or for `-- taskq:no-transaction` migrations the statements that remain applied plus any INVALID indexes found — and prints the single action to take. When the database connection itself cannot be established, there is nothing to diagnose, so it prints a short report naming the connection error and the same re-run action.
+
 **Example: apply all pending:**
 
 ```shell

docs/guides/upgrading.md

@@ -54,10 +54,85 @@ This is a deliberate tradeoff, not a missing feature:
    `{schema}.schema_migrations` are skipped. See [cli.md](cli.md#taskq-migrate-up)
    for the full option reference (`--phase`, `--target`, `--max-steps`).
 
+## Non-transactional migrations
+
+By default every migration file runs inside its own transaction, so a failure
+rolls the whole file back. PostgreSQL forbids some statements inside a
+transaction block — notably `CREATE INDEX CONCURRENTLY` and
+`DROP INDEX CONCURRENTLY`, the only forms that build or drop an index without
+blocking writes on the table. On hot tables (`jobs`, `job_events`) a plain
+`CREATE INDEX` takes a `SHARE` lock that blocks `INSERT`/`UPDATE`/`DELETE`
+for the duration of a full-table scan and stalls the worker fleet, so index
+migrations on those tables should use the concurrent forms.
+
+A migration opts out of the transaction wrapper with a header directive in its
+leading comment block (`--` line comments only, before the first SQL token):
+
+```sql
+-- taskq:no-transaction
+-- NOT redundant with IF NOT EXISTS below: an interrupted CREATE INDEX
+-- CONCURRENTLY leaves an INVALID index that IF NOT EXISTS alone would
+-- silently skip rebuilding, so drop the debris first.
+DROP INDEX CONCURRENTLY IF EXISTS "{schema}".jobs_queue_idx;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS jobs_queue_idx ON "{schema}".jobs (queue);
+```
+
+The runner then executes the file **statement by statement, each in its own
+implicit transaction** (the same semantics as Alembic's `autocommit_block` or
+Rails' `disable_ddl_transaction!`). This changes the failure contract, so
+three rules apply:
+
+- **The migration must be idempotent and re-runnable.** Nothing rolls back: if
+  the third statement fails, the first two stay applied. The ledger records
+  the migration only after *every* statement succeeds, so the next
+  `migrate up` re-executes the whole file — every statement must tolerate
+  being re-run (`IF NOT EXISTS`, guarded inserts, etc.).
+- **An interrupted `CREATE INDEX CONCURRENTLY` leaves an `INVALID` index
+  behind.** The standard remedy is drop-and-rebuild, written into the
+  migration itself as shown above: the `DROP INDEX CONCURRENTLY IF EXISTS`
+  line removes debris from an interrupted attempt before rebuilding. A plain
+  `CREATE INDEX CONCURRENTLY IF NOT EXISTS` alone would silently skip the
+  rebuild while the invalid index keeps its name. You never have to find
+  these by hand: when a run fails, `taskq migrate up` lists any INVALID
+  indexes in its failure report.
+- **No transaction-control statements.** `BEGIN`/`COMMIT`/`ROLLBACK` (and
+  aliases) are rejected before anything executes — they would silently
+  re-open a transaction, defeating the directive. The statement splitter
+  assumes the server default `standard_conforming_strings=on`.
+
+Operators can see the distinction two ways: `taskq migrate status` annotates
+non-transactional migrations with `(no transaction)`, and the
+`{schema}.schema_migrations` ledger records how each migration ran in its
+`use_transaction` column (`false` = ran outside a transaction). The runner
+adds that column when recording the next migration, so deployments upgraded
+from older TaskQ versions need no manual step; rows applied before the column
+existed read `true`.
+
 ## If a migration goes wrong
 
-- Stop workers pointed at the affected schema to avoid further writes.
-- Restore the database from the pre-migration backup.
-- Pin `taskq-py` back to the previous version until the issue is resolved,
-  since the previous version's code may not be compatible with the new
-  schema.
+`taskq migrate up` diagnoses its own failures: it tells you which migration
+failed, what state the schema is in, and the one action to take. You never
+need to inspect catalog state by hand.
+
+### Transactional migration (the default)
+
+The whole file rolled back automatically, so the schema is exactly as it was
+before the attempt. Fix the cause of the error, then re-run `taskq migrate
+up`.
+
+### Non-transactional migration (`-- taskq:no-transaction`)
+
+Nothing rolls back: statements before the failure remain applied, and the
+migration is **not** recorded. Re-run `taskq migrate up` — the migration is
+idempotent, and the command's failure report lists any INVALID indexes the
+interrupted attempt left behind; the drop-and-rebuild already written into
+the migration cleans them up on the re-run. Only pin `taskq-py` back to the
+previous version if the migration SQL itself is wrong and you need time to
+ship a correction.
+
+### A migration applied successfully but broke older workers
+
+Restoring from backup is for this scenario: the migration itself succeeded,
+but not-yet-upgraded workers cannot run against the new schema. Stop the
+workers pointed at the affected schema, restore the pre-migration backup,
+and pin `taskq-py` back until every worker is upgraded.

pyproject.toml

@@ -238,6 +238,9 @@ ignore = [
 "tests/test_idempotency_scope_migrations.py" = ["S608"]
 # Same rationale — schema name validated against _IDENT_RE; all
 # user-supplied values use $N parameter binding.
+"tests/test_migrations_populated.py" = ["S608"]
+# Same rationale — schema name validated against _IDENT_RE; all
+# user-supplied values use $N parameter binding.
 # SIM117: conn.transaction() depends on conn from pool.acquire();
 # async-with nesting cannot be flattened.
 "tests/test_singleton.py" = ["S608", "SIM117"]