Skip to content

perf(server): enable WAL and NORMAL sync for the SQLite store - #3543

Open
n1hility wants to merge 1 commit into
NVIDIA:mainfrom
n1hility:3494-sqlite-wal/jg
Open

n1hility wants to merge 1 commit into
NVIDIA:mainfrom
n1hility:3494-sqlite-wal/jg

Conversation

@n1hility

Copy link
Copy Markdown

Summary

On-disk SQLite stores ran with sqlx defaults (rollback journal, synchronous=FULL), so every autocommit write paid several fsyncs and blocked readers. openshell forward service does two such writes per forwarded TCP connection (session token minted and revoked), which made connection setup through a forward linear in the number of concurrent connections and pushed bursts into the per-sandbox connection cap. This switches on-disk stores to WAL with synchronous=NORMAL.

Related Issue

Fixes #3494

Changes

  • SqliteStore::connect: for on-disk URLs, switch the file to journal_mode=WAL once on a single connection before the pool opens (entering WAL needs exclusive access; done up front so pool connections only ever re-apply the pragma to a file already in WAL mode and a failure surfaces as one clear connect error), then build the pool with journal_mode=WAL and synchronous=NORMAL. In-memory databases are unchanged. A failed switch reports the file path and the exclusive-access requirement.
  • Tests (persistence/tests.rs): fresh on-disk store reports wal and synchronous=1 and its -wal/-shm sidecars are 0600; an existing rollback-journal database is switched on connect; concurrent readers proceed under a burst of insert-then-update writes on a file-backed store; the stale comment about non-WAL production is reworded.
  • Docs: architecture/gateway.md (durability trade-off, backup with sqlite3 .backup/VACUUM INTO, local filesystem requirement), docs/reference/gateway-config.mdx, deploy/rpm/CONFIGURATION.md, Helm values.yaml/README note that the SQLite volume must be local block storage.

Out of scope, noted for follow-up: an explicit store close on gateway shutdown for a final checkpoint; minting one session token per forward process instead of per connection; making the per-sandbox forward connection cap configurable.

Testing

  • cargo fmt -p openshell-server -- --check, cargo clippy -p openshell-server --all-targets --features test-support -- -D warnings, python3 scripts/update_license_headers.py --check all clean on the rebased branch (main @ the base of this PR).
  • cargo test -p openshell-server --features test-support persistence: 85 passed, 0 failed, 3 pre-existing ignores; the three new tests were also repeated 40 times with --test-threads=8 without a failure.
  • mise run pre-commit / mise run ci: not run locally (no mise on this machine); the equivalent fmt, clippy, license and unit checks above were.
  • Unit tests added/updated
  • E2E tests: not applicable to this change (no e2e path exercises store journal mode); measured behaviour instead:

Before/after with the same source, gateway on a hosted runner with the Docker driver, a sandbox serving loopback HTTP, openshell forward service in front, N simultaneous connections each doing one request (reproducer and scripts: https://github.com/n1hility/OpenShell/tree/forward-sweep-repro, workflow forward-sweep.yml; details on #3494):

simultaneous connections before: wall / completed after: wall / completed
6 0.058 s / 6 0.009 s / 6
16 0.110 s / 16 0.022 s / 16
32 0.135 s / 32 0.056 s / 31
64 0.337 s / 47 0.089 s / 53

On a 2 vCPU VM with SQLite on a network block volume the same sweep went from 1.52 s to 0.06 s at 16 connections and from 6.05 s to 0.12 s at 64. Remaining refusals above 20 connections are the fixed per-sandbox cap, independent of this change.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

On-disk SQLite stores ran with sqlx defaults: rollback journal
(`journal_mode=delete`) and `synchronous=FULL`. Every autocommit write paid
several fsyncs and blocked readers while it held the lock, so gateway hot
paths made of many small writes serialized on disk latency. The clearest
case is `openshell forward service`, which mints and revokes an SSH session
token around every forwarded TCP connection: two commits per connection,
tens of milliseconds each on a virtual disk, wall clock linear in the
number of concurrent connections, and enough queueing that bursts hit the
per-sandbox connection cap and get refused.

Switch on-disk databases to WAL with `synchronous=NORMAL`. The mode change
runs once on a single connection before the pool opens: entering WAL needs
exclusive access to the file, so doing it up front means pool connections
only ever re-apply the pragma to a file already in WAL mode, and a failure
surfaces as one clear connect error. The first start after upgrading an
existing database therefore needs the file to be otherwise unopened.
`synchronous` is applied through the connect options on every pooled
connection. In-memory databases keep their defaults. A crash can now roll
back the most recent transactions without corrupting the database, which
is the standard WAL trade-off and fits the single-node scope of the SQLite
backend.

Tests cover a fresh store, an existing rollback-journal file that must be
switched on connect, sidecar permissions, and concurrent readers under a
burst of insert-then-update writes. Architecture, configuration and Helm
docs describe the durability trade-off, the sidecar files, and the local
filesystem requirement.

Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@n1hility

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(server): forward service serializes connections on the SQLite store (two commits per TCP connection, rollback-journal mode)

1 participant