README.md

ubgo/workflow documentation

Guides for github.com/ubgo/workflow: one Go API for background jobs, cron schedules and multi-step workflows, on an in-memory driver, a SQL driver (SQLite, Postgres, MySQL 8), a Redis driver or the Hatchet engine. Every guide is step by step, ends with its pitfalls, and copies its Go code from a runnable program under examples/, which go test runs, so the code on these pages compiles and works.

The diagram numbers the two paths: steps 1 to 5 are an enqueue, from your app through the capability check and enqueue middleware to the stored run; steps 6 to 10 are its execution, from a worker claiming the run through step middleware to your step function and the run-finished observers. Your app depends on workflow.Client alone. Plugins, middleware and failover wrap a client without hiding what it can do, and each driver declares its capabilities, so a feature a backend cannot honour is refused at start-up instead of silently dropped (The client, Capabilities).

Contents

• Learning path
• Start from your goal
• All pages: Start, Choosing, Concepts, Jobs, Schedules, Workflows, Events, Drivers, Middleware, Observability, Batches, Reliability, Testing, Operations, Admin, Extending, Reference

Learning path

Read in this order to go from a first job to a production setup. Each step builds on the one before; skip what your app does not need.

#	Step	What you learn	Guide	Example
1	Run one job	Register a workflow, start the workers, enqueue a job, on the memory driver	Quickstart	quickstart
2	Write a real workflow	Steps that depend on each other, parallel steps, a join, retries, a permanent failure and its OnFailure handler	Your first workflow	first-workflow
3	Choose a driver	Memory, SQL or Hatchet, what each can do, and how the library refuses what a driver lacks	Choose a driver, Capabilities	capabilities
4	Control how jobs run	Retries and backoff, delayed jobs, cancel and status	Retries, Run a job later, Cancel and status	retries, delayed-jobs
5	Never run the same thing twice	Idempotency keys, concurrency limits per key, rate limits	Idempotency keys, Concurrency and rate limits	idempotency, concurrency
6	Schedule recurring work	Cron on a workflow and at run time, time zones and DST, missed ticks	Cron schedules, Time zones, Misfire policies	cron-schedules
7	Wait for the outside world	Wait steps released by Signal, deadlines, durable sleeps	Wait for an event, Sleep inside a workflow	signals-and-sleep
8	Test your app	The Fake that refuses what production refuses, assertions, the Recorder, a fake clock	Test app code with the Fake, Drive time in tests	app-tests
9	Make it durable	The SQL driver on SQLite, migrations run by your deploy	The SQL driver, Migrations	sqlite-worker, migrations-export
10	Add cross-cutting behaviour	Logging, recovery, timeouts, a circuit breaker, encrypted payloads, tracing and metrics, run-end observers	Step middleware, Payload codecs, OpenTelemetry, React when a run finishes	middleware-stack, opentelemetry, run-finished
11	Keep jobs and data consistent	Enqueue inside your transaction, the outbox, unique jobs, debounce	Transactional enqueue, Outbox, Unique jobs	transactional-enqueue, outbox, unique-jobs
12	Run groups of jobs	Batches with Then, Catch and Finally, progress, a durable batch store	Batches, Batch durability	batches, batch-durable
13	Go to production	Several workers on one database, clean shutdown, retention, failover, the launch checklist	Going to production, Deploy workers, Retention, Failover	postgres-cluster, retention, failover
14	Extend it	A plugin, your own store, your own driver held to the conformance suite	Write a plugin, Write a driver	custom-plugin, custom-driver

Start from your goal

I want to	Read	Example
Send an email after sign-up without blocking the request	Enqueue a job	quickstart
Pick a backend for my app	Choose a driver, Choose by kind of app	capabilities
Retry a flaky API call, honouring its Retry-After	Retries	retries
Send a reminder tomorrow at 09:00, and cancel it if the user acts first	Run a job later, Cancel and status	delayed-jobs
Run a nightly job, or a report per tenant in the tenant's zone	Cron schedules, Time zones	cron-schedules
Accept a webhook twice and process it once	Idempotency keys	idempotency
Keep one tenant from taking every worker	Concurrency and rate limits	concurrency
Pause a run until a payment or an approval arrives	Wait for an event	signals-and-sleep
Save an order and enqueue its job atomically	Transactional enqueue, Outbox	transactional-enqueue, outbox
Allow one export per user at a time	Unique jobs	unique-jobs
Reindex once after a burst of edits	Debounce	debounce
Process 10,000 rows and know when all are done	Batches	batches
Send payloads too big for the queue	Claim check	claim-check
Keep personal data out of the queue tables	Payload codecs	encrypted-payloads
Trace a job from the request that enqueued it	OpenTelemetry	opentelemetry
Keep accepting jobs while the engine is down	Failover	failover
Secrets in payloads	SecretGuard refusing credentials at enqueue, SecretRef resolved in the step
Run several worker machines on one Postgres	Deploy workers	postgres-cluster
Run my jobs on the Redis I already have	The Redis driver	redis-worker
Run my jobs on Hatchet	The Hatchet driver	hatchet-worker
Hybrid placement	Crons fire on one client, runs live on another; what the split refuses; several nodes
Test the code that enqueues, without running workers	Test app code with the Fake	app-tests
Send slow imports to their own workers and pause them during maintenance	Named queues	named-queues
See failed runs, retry one from the failed step, or clear a backlog	Manage runs, Admin operations	manage-runs, admin
Run a schedule only in office hours, and ping a heartbeat when it ends	Schedule conditions, Schedule hooks and pings	schedule-hooks
Send a confirmation email only if the request that ordered it succeeded	Deferred dispatch	deferred-dispatch
Tell a run waiting to retry from one waiting for a worker, and a dropped run from a failed one	Run states	run-states
Let a job decide what runs after it	Chains of jobs	chain-edits
Memoise work inside a step	Memo and the journal, MapEach, Loop, ProcessChunks with progress, run data
Branching	Choice: one branch picked at run time, built from ordinary steps
Child runs	Spawn and AwaitChildren: children started once per run, waiting without holding a worker

Events

Page	What it covers
Events	What the events package is and is not, quick answers
Publish and subscribe	A Bus, subscriptions and filters, deduplication by event ID, delivering to waiting runs
Inbound webhooks	NewWebhook and its status codes, HMAC signatures, Standard Webhooks, your own verifier
Give a backed-up queue the workers an idle one does not need	Worker balancing
Run a small Go function in the background without registering a workflow	Closures
Run a small Go function in the background so it survives a restart	Functions by name
Typed payloads	JobOf, DecodePayload, payload size limits, payload versions with upcasters
Progress and heartbeats	ReportProgress, RunInfo.Progress, progress events, Heartbeat and Step.HeartbeatTimeout
Results and awaiting a run	A run's result, Await, AwaitResult, RunFailedError
Timeouts	The four deadlines, Workflow.RunTimeout, OnTimeout, the ErrTimeout family
Tenants	Job.Tenant, reading it in steps and events, RunQuery.Tenant
Upstream rate limits	A leaky-bucket, cost-based limit on calls to an outside API, Observe, your own bucket store
Keep what a nightly task logged in a file, and mail it when the task fails	Task output
Restart a worker after so many jobs, so long, or above a memory size	Worker recycling
Nodes and health	Which node ran each attempt, the online nodes, liveness and readiness probes, scheduler-only and worker-only processes
Keep a slow hourly job from starting again while it still runs, and run it now by hand	Overlap, trigger now and last fire
Local cron	An in-process cron that runs on every node: cache warmers, local cleanup
Be told once when runs keep failing, and send their errors to Sentry	Failure alerts and error reporting
Lifecycle events	A live feed of inserts, attempts, retries, progress and ends; best effort, local; how it differs from the run-finished hook
Stuck jobs	Report an attempt that runs far too long without touching it; Stuck vs Timeout vs HeartbeatTimeout
See every attempt's error, or end a stuck run by hand	Manage runs › Error history, Force a run to an end
Let web servers enqueue without running workers	Deploy workers › Web nodes that only enqueue
Start jobs at once on Postgres instead of at the next poll	The SQL driver › Wake workers at once
Give operators a dashboard API with my own login	Admin JSON API
Give operators a web dashboard behind my own login	Admin web UI
Find every run of one customer, or every order number starting with A-	Search by label
Retry a step without charging the card twice	Memoise work inside a step
Accept a signed webhook and start a run once per delivery	Inbound webhooks
Stay inside an outside API's call limit per account	Upstream rate limits
Keep tokens and passwords out of stored payloads	Secrets in payloads
Wait in a step until a third-party export is ready, without spending retries	Retries › Snooze
Show a progress bar for a long export and hand back its result	Progress and heartbeats, Results and awaiting a run
Change a job's payload shape without breaking queued jobs	Typed payloads
Show a live progress bar and every retry as it happens	Lifecycle events
Give Kubernetes a liveness and a readiness probe	Nodes and health
Fire crons in my process and keep the jobs in the database	Hybrid placement
Fire crons on one chosen node, or keep nodes with drifting clocks from stealing each other's steps	The SQL driver › Clocks, skew and one fire node
Retry every failed run of a workflow from a terminal	The workflow command
Run a schedule only in production, or only during a campaign week	Schedule conditions
Warm a cache on every node every minute	Local cron
Check whether the library has a feature	Feature index

All pages

Start

Page	What it covers
Install	The core module, the separate modules (sqlstore, redis, hatchet, otel, sentry, admin, cli) and the Go version each needs
Quickstart	One workflow on the memory driver in about thirty lines, then the same code on SQLite
Your first workflow	Four steps with a retry, a parallel pair, a join and a failure handler
Going to production	The checklist: durable driver, migrations, idempotent handlers, clean shutdown, retention, logs

Choosing

Page	What it covers
Choosing	The decision guides, with quick answers to the first decisions
Choose a driver	Memory, SQL or Hatchet: durability, machines, what you operate, capabilities
Choose a method	Enqueue, EnqueueTx or the outbox; cron or Schedule; batches; duplicate guards; limits; delays; retries; run-end hooks
Choose by kind of app	A starting configuration for nine kinds of app

Concepts

Page	What it covers
The client	The one Client interface, start and stop, validation
Capabilities	What a driver declares, checking workflows and schedules, refusals
Errors	Sentinel errors and the backend-outage marker
Labels	Job labels and the reserved prefixes
Plugins	Adding a feature to a client with workflow.Use
Time	The injected clock every scheduling decision reads

Jobs

Page	What it covers
Enqueue a job	Job, RunRef, payloads, labels and priority
Run a job later	Job.RunAt and scheduled runs
Idempotency keys	One run per key, and requiring or ignoring keys per workflow
Retries	Retry policies, per-step retries, backoff and jitter, Retry-After, permanent errors, snooze
Concurrency and rate limits	Limits per key, strategies for the run over the limit, FIFO per key, rate limits
Cancel and status	Stopping a run and reading where it is
Named queues	Workflow.Queue and Job.Queue, workers, priority and weights per queue, pausing a queue
Manage runs	List, search by label, retry, delete and bulk-cancel runs; due-time filters, reschedule, cancel one step; pause a workflow, a queue or everything
Run a workflow inline	RunInline: one run in the caller's goroutine, with no client or workers
Run states	The retrying and discarded statuses, next-attempt time, querying them, batches and discarded jobs, switching on statuses
Deferred dispatch	Jobs enqueued during a request, sent only when it succeeds: DeferHTTP, Deferred by hand, and when a transaction is better
Closures	A Go function run as an in-process job on the memory driver, and why durable drivers refuse it
Functions by name	A Go function registered under a name and enqueued with a JSON argument: durable closures on every driver

Schedules

Page	What it covers
Cron schedules	Cron on a workflow and at run time, the syntax, a seconds field, calendar specs, parse errors, other dialects, one fire across nodes, the tick a run is for
Time zones	Schedules in a zone, loading zones by name, the embedded zone database, next runs in two zones, daylight saving time
Misfire policies	What happens to ticks missed during downtime
Schedule conditions	Time-of-day windows and ScheduleWhen, environment gating, an active period, the answer to between(), unlessBetween(), when() and skip()
Schedule hooks and pings	Tick events, per-schedule hooks and heartbeat pings
Manage schedules	List schedules with their next fire, pause and resume, group schedules, keep the newest runs, store a schedule as JSON
Task output	Capture what a task logs and returns and send it to a file, a writer, a mail function or the database
Overlap, trigger now and last fire	Skip a tick while the previous run is unfinished, run a schedule now, read its last fire and run

Workflows

Page	What it covers
Workflows and steps	Defining a workflow, typed handlers, dependencies, step info, timeouts, failure handlers
Wait for an event	Wait steps, matching events to runs, deadlines
Sleep inside a workflow	Durable sleep steps
Chains of jobs	Job.Chain, editing the rest of a chain from a running step, failure, and child runs on Hatchet

Drivers

Page	What it covers
The in-memory driver	For tests, local runs and work that may be lost
The SQL driver (sqlstore)	SQLite, Postgres and MySQL 8: workers, polling, wake-ups with LISTEN/NOTIFY, leases, database errors, run IDs
The Redis driver	Durable runs and cron on Redis: whole-run leases, fenced writes, what it still refuses, shared stores, deployment, upgrading
The Hatchet driver	Configuration, what it refuses, engine quirks, the native SDK

Middleware

Page	What it covers
Step middleware	Recover, timeout, fail on chosen errors, skip, circuit breaker
Enqueue middleware	Wrapping every enqueue, routing jobs
Payload codecs	Transforming payloads, encryption with key rotation

Observability

Page	What it covers
Logging	A log line per step attempt, and driver logs
OpenTelemetry	Spans, trace context from enqueue to step, parenting, span names, labels, metrics
React when a run finishes	The run-finished hook, and its best-effort form on Hatchet
Failure alerts and error reporting	Alerts on final failures with dedup and rate limits, error reporters, Sentry

Batches

Page	What it covers
Batches	Dispatch with Then, Catch and Finally, progress, cancel, labels, IDs
Batch completion	How a batch learns a job ended, Hatchet's source, retrying callbacks
Batch durability	Where batch state lives, the SQL store, resuming a dispatch
Native group runs	Bulk triggers on Hatchet, and the batch coordinator
Change a batch after dispatch	Before and Progress callbacks, Add, RetryFailed, Prune, EnqueueMany
Wait for a batch in a workflow	AwaitStep: a step that dispatches a batch and waits for it

Reliability

Page	What it covers
Unique jobs	Unique for a time, until the run finishes, by workflow and arguments, by tagged fields, by holder state, returning the existing run, across processes
Debounce	One run per burst, with shared state
Claim check	Big payloads and outputs in a blob store, deleted when the run ends
Transactional enqueue	EnqueueTx inside your own SQL transaction
Outbox	Jobs written in your transaction and relayed to any client
Failover	A second backend for enqueues, outage classification, cool-off

Testing

Page	What it covers
Test app code with the Fake	A client that runs nothing, assertions, matchers, operator actions, behaving like production
Record a real client	Recording the enqueues of a real driver
Drive time in tests	A fake clock for delays, retries, sleeps and crons
Test batches, chains and a subset of workflows	Batch and chain assertions, FakeExcept and FakeOnly
Test one step	RunStep and its outcomes: succeeded, failed, permanent, discarded, released
Test queues, retries and pauses	AssertEnqueuedOn, AssertRetried, AssertDeleted, AssertPaused
Run everything now in tests	Memory RunAll: what it hurries (delays, backoffs, snoozes, sleeps) and what it does not (timeouts); ErrRunAllStalled

Operations

Page	What it covers
Deploy workers	Several workers on one database, web nodes that only enqueue, clean shutdown without spending a retry
Migrations	The ledger, each store's migrations, rollbacks, your own migration tool, table names
Retention	Keeping, pruning and archiving finished runs
Tuning	Housekeeping and Hatchet limits
Worker balancing	Moving workers between queues by depth or wait, bounds, the balance loop, your own strategy
Worker recycling	Stop a worker after a number of jobs, a time or a memory size so its supervisor starts a fresh one

Admin

Page	What it covers
Admin operations	The admin module's Go operations over any driver: runs, failures, retry, delete, cancel, force, bulk actions with dry run, reschedule, cancel a step, pause, schedules with trigger-now, queue stats, captured output, run-end charts, nodes and health
Admin JSON API	A closed-by-default HTTP API with typed Identify and Authorize hooks
Admin web UI	The dashboard the admin handler serves: screens (a run's attempts, output, progress and result, a schedule with Run now, charts, system health), cookie login, CSP, keys
Queue alerts	admin.Monitor thresholds on queue depth and wait, and the OpenTelemetry depth gauge
The workflow command	The CLI over the admin API: auth flags and env vars, every command, --json, --dry-run, exit codes, embedding

Extending

Page	What it covers
Write a driver	A complete driver under 300 lines, held to the conformance suite
Write a plugin	What a plugin can provide, and a tenancy plugin
Write middleware	Step and enqueue middleware of your own
Custom stores	Your own store for unique locks, debounce, circuits, blobs or batches

Reference

Page	What it covers
Feature index	Every feature, with its package, guide section and example
Capabilities	Every capability and which driver has it
Errors	Every sentinel error, which call returns it and what to do
Benchmarks	Enqueue and end-to-end throughput and pickup latency per driver, method, caveats and how to rerun
Names	Reserved labels, tables, migration owners, Hatchet metadata, IDs, span and metric names

The runnable programs are listed in examples/README.md.