batch_id exists only as a key inside jobs.metadata — there's no batches table, and no list_batches-style call. BatchHandle carries the id, but it's an in-memory object that dies with the process that enqueued the batch.
The consequence is that batches are addressable but not discoverable. JobFilter(batch_id=...) works well once I have the UUID; there is no way to ask "which batches exist on this queue", or "which batches are still outstanding", because the only place the id is recorded is spread across the rows it points at, and JobFilter can't project distinct metadata values.
That bites on restart. A supervisor that comes back after a crash and wants to resume oversight of in-flight fan-outs has nothing to enumerate. The two ways out are both workarounds for the missing introspection rather than solutions:
- persist every
batch_id in my own table as I create it, so I can read them back later — duplicating state the jobs rows already contain;
- or make batch ids deterministic, e.g.
uuid5(ns, f"{run_id}:{stage}"), purely so they can be re-derived without being stored. This is what I settled on, and it works, but "derive the id so I never have to look it up" is a strange thing to be pushed into by a library that already has the rows.
What I'd rather write:
batches = await client.batches.list(JobFilter(queue="ingest", active=True))
# → [(batch_id, BatchCompletionStatus), ...]
or, more minimally, a way to get the distinct batch_ids matching a JobFilter, which is enough to bootstrap into the existing BatchHandle.status / wait_for_batch surface. The GIN index on metadata is already there; batch_id is already a first-class JobFilter field and a typed UUID on BatchHandle, so it's the one metadata key the library already treats as structural.
Open questions:
- Does a real
batches table earn its keep (created_at, expected size, originating actor, a place to hang a failure policy), or is enumeration-over-jobs sufficient? A table would also give the expected-count discussed in the fan-out-then-finalize hazards somewhere to live.
- If batch records became first-class, they'd need to prune alongside their jobs — presumably following the same succeeded/failed retention as
jobs → jobs_archive, but a batch outliving its pruned children raises the question of what its status even means.
- If "derive deterministic ids" is the intended pattern, that's worth stating in the batch docs, because nothing currently points newcomers at it and the natural reading of
batch_id=None auto-generating a UUIDv7 is that you're meant to keep the handle.
batch_idexists only as a key insidejobs.metadata— there's no batches table, and nolist_batches-style call.BatchHandlecarries the id, but it's an in-memory object that dies with the process that enqueued the batch.The consequence is that batches are addressable but not discoverable.
JobFilter(batch_id=...)works well once I have the UUID; there is no way to ask "which batches exist on this queue", or "which batches are still outstanding", because the only place the id is recorded is spread across the rows it points at, andJobFiltercan't project distinct metadata values.That bites on restart. A supervisor that comes back after a crash and wants to resume oversight of in-flight fan-outs has nothing to enumerate. The two ways out are both workarounds for the missing introspection rather than solutions:
batch_idin my own table as I create it, so I can read them back later — duplicating state the jobs rows already contain;uuid5(ns, f"{run_id}:{stage}"), purely so they can be re-derived without being stored. This is what I settled on, and it works, but "derive the id so I never have to look it up" is a strange thing to be pushed into by a library that already has the rows.What I'd rather write:
or, more minimally, a way to get the distinct
batch_ids matching aJobFilter, which is enough to bootstrap into the existingBatchHandle.status/wait_for_batchsurface. The GIN index onmetadatais already there;batch_idis already a first-classJobFilterfield and a typedUUIDonBatchHandle, so it's the one metadata key the library already treats as structural.Open questions:
batchestable earn its keep (created_at, expected size, originating actor, a place to hang a failure policy), or is enumeration-over-jobs sufficient? A table would also give the expected-count discussed in the fan-out-then-finalize hazards somewhere to live.jobs→jobs_archive, but a batch outliving its pruned children raises the question of what its status even means.batch_id=Noneauto-generating a UUIDv7 is that you're meant to keep the handle.