[Spec #130] Sync multi-machine emdash - #145
Merged
Merged
Conversation
Owner
Author
Review — 7 area reviewers + adversarial verification (2026-08-10)Reviewed at Reassurance, with one correction: K0 never transits at join (verified end-to-end — only a SHA-256 of the join half reaches the relay). But at space creation the relay Worker currently mints K0 and returns it in the secret ( ✅ Fixed and pushed (
|
Owner
Author
Fix pushed to make it testable (
|
…for multi-machine sync Hand-written migration 0025 (spec #130, ticket #132): - projects.path becomes nullable with the unique index on path preserved (SQLite treats NULLs as distinct, so multiple NULL paths are legal). - The runner executes with foreign_keys=ON inside one transaction, so a naive rebuild would cascade-delete children. The rebuild instead copies projects + every referencing table (and their own children, messages and automation_runs) to _new tables, drops the originals, renames into place and recreates all indexes. Verified empirically against sqlite 3.51. - Portable tables (projects, project_settings, project_remotes, tasks, conversations, automations) gain a sync_ts INTEGER (ms) clock, backfilled with the migration-time epoch and maintained by AFTER INSERT/UPDATE triggers (trg_<table>_sync_ts_ins/upd) for push detection (WHERE sync_ts > lastPushed). app_settings/kv stay kv-style without a clock; the sync engine handles their keys selectively. - Machine-local device identity (device:id, device:name KV namespace) and reserved app_secrets keys for the sync credential and encryption key are added; nothing is wired into behaviour yet. Migration tests seed project children (tasks, conversations, project settings, remotes, terminals, editor buffers, messages, automation runs) and assert preservation, FK integrity, NULL-path handling and the sync clock triggers.
… for multi-machine sync Machine-local device identity lives in the kv table under the device: namespace (device:id UUID, device:name hostname), deliberately separate from the telemetry instanceId. app_secrets keys sync-token and sync-encryption-key are reserved for the sync engine (ticket #133) via the encryptedAppSecretsStore idiom; nothing wires them into behaviour yet. (spec #130, ticket #132)
Commit 6c24c70 added sync_ts (integer NOT NULL DEFAULT 0) to projects, tasks and conversations in src/main/db/schema.ts. Drizzle includes the column in generated INSERT statements (it has a default), so the legacy-port importer tests, which build a minimal destination schema by hand, failed with 'table projects has no column named sync_ts' — 8 tests across service.test.ts and relational.test.ts. Mirror the production schema by adding the sync_ts column to the hand-written projects/tasks/conversations DDL in both test files.
New @emdash/sync-relay workspace package: the server side of the multi-machine sync spec (#130). The relay is an ordering relay for two personal devices: it stores opaque, client-encrypted row bodies plus plaintext metadata and never parses body content. - Space & devices: POST /v1/space (create space, first device token, pairing secret), POST /v1/join (pair a second device; single-use, 15-minute TTL, per-secret 5-attempt budget enforced in the Worker), GET /v1/devices, POST /v1/devices/revoke, POST /v1/devices/join-secret (existing device mints a fresh pairing secret). - Sync: POST /v1/sync/push stamps a per-space monotonic version via INSERT ... ON CONFLICT DO UPDATE ... RETURNING version inside the same db.batch() transaction as the row upsert; last-write-wins by server receipt order, stale pushes accepted, never rejected. POST /v1/sync/pull returns rows with version > cursor, ordered; tombstones are rows with deleted=1. POST /v1/sync/poll is the long-poll channel (re-check D1 each second, timeout clamped to 25s). - Auth: bearer device tokens (32 random bytes base64url, prefix + checksum verified with a constant-time comparison); only SHA-256 of the token is stored; revoked tokens are refused; every query is scoped to the token's space. - Schema bootstraps idempotently at startup: spaces, tokens, join_secrets, sync_rows (indexed on (space_id, version)), version_counters. - Tests: in-process D1-compatible harness on node:sqlite (no workerd download), 35 tests covering the version counter, cursor semantics, LWW, tombstone supersession, opaque body passthrough, token auth, revocation, space scoping, and the pairing attempt budget. - wrangler.jsonc with the D1 binding plus a README documenting the protocol, storage, security notes, and how the fork owner deploys it. No secrets, no CI deploy; deployment is manual. Refs: #131
readJson previously passed JSON null and primitives through to service functions, where property access threw a TypeError and surfaced as a 500. Normalize to a 400 at the request boundary, and default a non-string device name in createSpace instead of crashing.
…city, poll wait path
…ecreate The 0026 INSERT triggers stamped plain wall-clock ms, so a delete plus a same-millisecond re-create of a row could reproduce the clock recorded in sync_row_state (or equal the per-table watermark). The engine's "applied-untouched" guard then mistook the new row for an untouched copy and never pushed it (or the sync_ts > lastPushed filter excluded it), so resurrections and delete+recreate cycles were dropped intermittently. INSERT stamps are now MAX(wall clock, recorded row_sync_ts + 1, watermark + 1): a fresh row always outranks any clock the engine has seen for it and any watermark, while applied-untouched rows still match their recorded clock exactly and stay silent. Adds deterministic migration regression tests for both monotonicity guarantees and regenerates the fixtures against the final 0026 schema.
Also extends the raw-transport test to assert conversations.config and automations.trigger_config future-version blobs travel verbatim, not just tasks.linked_issue.
…batch A child upsert whose in-scope FK parent is absent locally (parent tombstone already applied here, or never present) aborted the whole pull transaction with a FOREIGN KEY violation. Because the relay replaces rows in place, the parent's creation upsert can be gone from the relay by the time a child edit from a machine that had not yet applied the tombstone lands — a fresh machine joining after a project delete pulled the carried project_remotes upserts with no project row, and a task edit racing a project delete wedged the deleting machine's sync permanently (cursor never advanced past the failing patch). Add importSkipIfMissingParent to the allowlist (tasks, conversations, automations, project_settings, project_remotes -> projects/tasks): such patches record the server version and are skipped (new skippedOrphan counter) so the pull cursor keeps moving; the deleting machine's cascade tombstone converges the row. Regression tests cover both scenarios.
Extend the repositoryWorkspaceId import test: a fresh import nulls the out-of-scope ssh_connections reference too, and each machine keeps its own ssh_connection_id when a remote row wins LWW (same treatment as path and repository_workspace_id).
…ials PairingService attaches this machine to a relay sync space (create/join), mints fresh pairing secrets for additional devices, and lists/revokes devices. The relay device token and space id are stored machine-locally in one safeStorage-encrypted app_secrets entry (SYNC_TOKEN_SECRET_KEY); the join credential is derived exactly as the relay verifies it (SHA-256 of the full emdj1_ secret). Errors from the relay's 4xx JSON bodies are mapped to typed PairingError codes with user-facing messages. Ticket #135.
The app registers itself as handler for the emdash URL scheme (macOS open-url, Windows/Linux second-instance argv) and forwards an emdash://join?secret=... link to the renderer, which opens the Settings Devices tab with the join modal pre-filled. The packaged builds declare the scheme via electron-builder protocols. Joining still requires explicit user confirmation in the modal. Ticket #135.
DevicesSettingsCard (following the SshConnectionsSettingsCard pattern) shows the not-paired state with create/join actions when this machine has no credential, and otherwise lists the space's devices with self/revoked badges and last-seen times. Add device mints a fresh single-use secret and displays it with a copy button, deep link, and 15-minute/single-use warning; revoking goes through the house confirmation dialog. Relay failures surface as user-facing toasts, never raw JSON. Ticket #135.
… and Devices UI - PairingService against a fake RelayAuthApi implementing the relay's pairing semantics: create/join/mint/list/revoke, single-use, TTL, and attempt-limit rejections, not_paired/unauthorized, and error mapping that never leaks raw relay JSON. - Join credential derivation verified against the relay's own crypto (apps/sync-relay/src/crypto.ts imported directly; the package entry is dist/index.mjs which the gate does not build). - Token storage round-trip through the real EncryptedAppSecretsStore with a fake safeStorage and fixture DB (main-db project). - Deep-link parsing/event forwarding with mocked electron. - Browser (Playwright) test for the Devices card: unpaired state, create-space secret modal, device listing, minting, toasts, and revoke confirmation. Ticket #135.
…y protocol The relay's join() (apps/sync-relay/src/service.ts) parses the presented join_hash as the raw emdj1_ secret — it needs the embedded space id and checksum — then compares SHA-256 of what the client sent against the stored digest. The client was pre-hashing the secret (sha256 hex), which the relay rejects with 401 'invalid join secret': pairing could never succeed end-to-end. deriveJoinHash now returns the trimmed secret itself; the join-credential tests exercise the real relay join service (createSpace -> derive -> join -> token, plus pre-hash and single-use rejection), and the pairing fake mirrors the relay's digest comparison.
Previously a paired machine whose listDevices call failed (relay unreachable) fell through to the 'Not paired with any device' branch, misleadingly offering to create/join a space. The paired+list-failure state now renders the error block with a Retry button. Browser tests cover the retry state and the deep-link pre-fill flow (modal opens pre-filled, nothing sent until the user confirms).
Retrofit the relay to the spec #130 two-half pairing model (#134): pairing secrets are now emdj1_<space>_<join_half b32>_<k0 b32> with only SHA-256 of the join credential stored (createSpace mints both halves, the join-secret endpoint registers a client-minted digest, join carries space_id and the bare credential). sync_rows gains client_version, stored verbatim so decrypting clients can bind it into the AES-GCM AAD.
Ticket #134. Bodies are encrypted in the main process before upload and decrypted on pull; the relay only sees plaintext metadata plus an opaque versioned envelope. - crypto.ts: CryptoHelper over node:crypto — HKDF-SHA256 per-row keys (salt = [table, pk], info 'row-v1'), AES-256-GCM with a fresh 96-bit nonce per encryption, envelope {alg, key_id, nonce, ct}, AAD = [table, pk, client_version, key_id]. key_id = first 8 bytes of SHA-256(K0). Two-half pairing secret format emdj1_<space>_<join b32>_<k0 b32> with the join credential = the bare base32 join half. - space-key-store.ts: K0 + key_id in safeStorage (app_secrets) under SYNC_ENCRYPTION_KEY_SECRET_KEY; rekey = new K0, new key_id. - encrypting-transport.ts: RelayTransport decorator — encrypts at push, decrypts on pull, flags undecryptable patches (decryptError) instead of throwing; missing key fails pushes with a clear error. - engine.ts: upserts carry client_version (last-known server version, 0 for new rows); patches with decryptError are recorded and counted as skippedUndecryptable, the pull continues. - pairing.ts/auth-api.ts/transport.ts: two-half pairing — createSpace and joinSpace extract and store K0, mintSecret composes the secret client-side (fresh join half + constant K0) and registers only the SHA-256 digest with the relay. - Client + relay tests cover the envelope format, per-row derivation, AAD tamper directions, decrypt failure modes, the two-client K0 invariant, mint-join flows, no-plaintext-on-the-wire, and pull continuation past undecryptable patches.
…esh key-store doc
…ate, auto-attach, merge, SSH re-attach) Ticket #136 (spec #130). A synced Project with no local anchor is surfaced as Unattached (distinct from path-not-found): local projects have a NULL path, SSH projects a NULL connection. Auto-attach scans the default projects directory for a repo with matching live remotes and attaches silently; manual attach re-anchors a local project to a picked directory (deduping against the unique path index by merging into an existing same-type project when remotes match, prompting on a local/SSH ambiguity) or re-attaches an SSH connection (fingerprint (host, port, username) + path, not connectionId). Engine: SSH remote paths now travel in the payload (local paths stay machine-local), an import path colliding with the unique index is nulled instead of wedging the pull, and an injectable projectAttachHook fires after fresh project imports for the auto-attach service. Attach re-runs ensureRepositoryWorkspace so tasks are provisionable on demand.
…k errors are swallowed (ticket #136)
… holds the picked attach path (ticket #136) The unique path index made the direct attach UPDATE throw an uncaught SQLITE_CONSTRAINT when the synced project carried no remotes (the remote-mismatch guard is skipped for empty synced remotes) and the picked directory belonged to another remotes-less local project. findMergeCandidates now treats a non-mergeable local holder at the picked path as a hard path-conflict, mirroring the SSH-holder branch.
…ry-instance workspaceId A repository-instance target stored in a synced automation task config must not carry a machine-local workspace reference (the workspace id belongs to the creating machine only). v3 makes workspaceId optional; v2 blobs keep their id through the upgrade chain. Producers emit version 3 configs. Issue: #138
…st the project workspace prepareCreateTask resolves a repository-instance target without a stored workspaceId from the mounted project's repository workspace. An Unattached project (no repository workspace on this machine) fails with a typed workspace-not-resolved error BEFORE any task row is committed; automation runs surface it as a project_unattached run failure instead of a crash or half-created task. Issue: #138
…nd v3 target storage
- automations.source ('local' | 'imported') is a machine-local column like
enabled: never transported in the sync payload; fresh imports are stamped
'imported' (importInsertColumns) while LWW conflict updates keep each
machine's own value. Migration 0027 adds the column with DEFAULT 'local'.
- The Automation type carries source; the automations list row and detail
sheet show an 'Imported, disabled' badge for fresh imports.
- The automation form now stores repository-instance workspace targets
without the machine-local workspace id (v3), resolved at run time.
Issue: #138
…0-day cap The spec (#130) assigns tombstone GC to the relay: record each device token's last-pull cursor and hard-delete a tombstone once every non-revoked device has pulled past its version, with a 90-day safety cap. The relay previously retained tombstones forever. - pull_cursors(space_id, token_id, cursor, updated_at): advanced (never rewound) on every pull that returns rows; a device that never pulled counts as behind everything, a revoked device never blocks collection. - gcTombstones() runs opportunistically on pulls that returned rows, deleting tombstones whose version is at or below the smallest active cursor or whose age exceeds the 90-day cap. - relay tests: retention until every active device has pulled, revoked devices not blocking, the age cap collecting behind devices, and resurrection at a fresh version after collection.
…e push) The spec (#130) says the sync service pushes on local writes (debounced) so both machines stay up to date without intervention, but the poll loop only ran a push+pull when the relay woke it with patches, on reconnect, or on manual sync. Edits made while the other machine was quiet sat unpushed until a relaunch or a manual Sync now. The poll cadence now doubles as the debounce: when a poll returns no patches and the engine reports pending rows, the loop runs a sync cycle instead of sleeping.
…loop safety Five confirmed defects from the PR review, all adversarially verified: - relay: single-use join was a TOCTOU race — read used_at, then wrote it in a separate round trip, so two interleaved /v1/join calls could both mint a token from one secret. consumeJoinSecret is now an atomic guarded UPDATE ... WHERE used_at IS NULL AND expires_at > now AND attempts_left > 0 RETURNING, and join() only mints when it wins the consume. +store-level test. - engine: project_remotes (initial-only) was gated on the parent project's first push, so a remote added after the project first synced (remotes are populated on task provision, not at creation) never synced at all — an empty auto-attach hint forever. Gate the carry per remote row (own row-state null) instead. Updated the initial-only test to the corrected per-row semantics and added a regression test (project synced with zero remotes, remote added later, carried once). - relay-config: EMDASH_SYNC_RELAY_URL unset silently defaulted to sync-relay.emdash.sh — upstream product infra the fork must not use for sync identity. Default to a reserved unresolvable .invalid host (fails fast, non-sync users unaffected) and expose `configured`. - automations: v3 stripped the machine-local workspaceId from EVERY repository-instance target, silently redirecting a 'use-existing' automation to the project root. Strip only for the 'repo-root' preset. - sync-service: the runLoop catch block called markPollFailure (async buildEngine + sqlite pendingCount) with no inner guard, and runLoop is fire-and-forget — a secondary throw killed background sync forever with no trace. Wrap the failure path and attach a last-resort .catch to loopPromise.
…nd machine A synced task arrived on the second machine carrying the origin machine's `tasks.workspace_id` verbatim — a reference to a per-machine `workspaces` row that never exists on the receiver (worktrees are machine-local, not synced), so the task was permanently unopenable (`ensureWorkspaceSetupForTask` → missing-workspace). This blocked the headline flow "pick up work on the other machine" (spec #130 stories 2 / 25, ticket #136). - allowlist.ts: `workspace_id` is now `importPreserveLocalColumns` (the same treatment `projects.repository_workspace_id` already gets) — never applied at import. A fresh import leaves it NULL; an already-provisioned local row keeps its own id across every later sync. (The reviewer-suggested `importNullIfMissingFk` would instead clobber the receiver's own workspace on every pull, orphaning its worktree — preserve-local is correct.) - workspace-bootstrap-service.ts: when a task has no local workspace and its project is attached, mint a local/SSH worktree row on demand (location/SSH derived from the project like createTask) and re-point the task; `config` is left NULL so the existing intent resolution falls back to the task's own branch (`use-branch`) and checks out the branch pushed from the origin machine. Until the project is attached there is nothing to provision against, so it still reports missing-workspace. Tests: bootstrap on-demand mint (main-db); workspace_id NULL-on-fresh-import / preserved-on-update convergence (sync-engine, main-db). node 3238, main-db sync+bootstrap green; typecheck, oxlint, oxfmt clean.
Proves the headline multi-machine flow with two independent client databases (two "machines") syncing through the actual relay code (apps/sync-relay service + store + schema over an in-process D1) — not a fake transport: - pairing derives one shared K0 from the pasted secret (createSpace on A, join on B, real relay functions); - a project + task created on A push through the real EncryptingRelayTransport (AES-256-GCM) and converge on B: task name, workflow stage and branch all arrive, and the project lands Unattached (its machine-local path never travels); - the relay stores only ciphertext — the plaintext task name never appears in sync_rows, and the stored body is a versioned AES-256-GCM envelope. Runs in the main-db vitest project. The relay service is imported directly (not via the HTTP handle() router) so no Cloudflare Workers types leak into this project's typecheck; token-auth resolution is covered by the relay's own suite.
The relay URL is public (workers.dev names are discoverable) and space creation / join are unauthenticated by design, so anyone who finds the URL could create spaces and push data — burning the operator's Cloudflare free-tier quota (self-DoS). The fork is public, so nothing secret can ship in the binary. Gate the relay with a pre-shared key the operator sets per machine by hand: - Relay (apps/sync-relay): every request must carry `X-Relay-Key`, checked in constant time (SHA-256 digests) before routing — including the space/join endpoints — else 401. `handle()` takes the expected key; the `fetch` entry point reads it from a `RELAY_KEY` Worker secret and FAILS CLOSED (500) when unset, so a forgotten secret never leaves the relay open. Existing 3-arg `handle()` test callers stay ungated (no behaviour change for them). - App: `SYNC_RELAY_CONFIG` reads `EMDASH_SYNC_RELAY_KEY` alongside the URL; both HTTP clients (HttpRelayTransport sync + HttpRelayAuthApi pairing) send `X-Relay-Key` when configured. `configured` now requires both URL and key. No default URL or key ships (public repo) — each machine is configured by hand; unset ⇒ unresolvable .invalid host, sync refuses to run. - README: deployment now sets `wrangler secret put RELAY_KEY`; documents the gate, the free-tier-quota rationale, the app env config, and self-hosting. The key gates the operator's infrastructure/quota; it is not a data secret (row bodies are already E2E-encrypted) and travels only over TLS. Tests: relay gate (401 without/with-wrong key, 200 with, ungated when unset), client sends the header when configured and omits it otherwise. relay 60, transport 6, node 3240, e2e 1; typecheck/oxlint/oxfmt clean both projects.
… env The relay URL and pre-shared key were env-only (EMDASH_SYNC_RELAY_URL / EMDASH_SYNC_RELAY_KEY), which is fine for dev but unusable for a packaged app (GUI apps don't inherit the shell env). Let each machine be configured by hand in the app, with env vars still overriding for dev/power users. - RelaySettingsStore: URL + key in one safeStorage `app_secrets` entry (machine-local, never synced — app_secrets is out of the sync allowlist), the same idiom as SyncCredentialsStore. The key is never returned to the renderer. - resolveRelayEndpoint: env → stored → unresolvable `.invalid` fallback, with `configured`/`envManaged` flags. Transports (HttpRelayTransport sync + HttpRelayAuthApi pairing) now take an async endpoint provider and resolve it per request, so settings entered in the app take effect without a restart. - sync RPC: getRelaySettings / setRelaySettings (validates https URL + non-empty key, kicks a sync) / clearRelaySettings. - UI: a "Sync relay" Settings card (URL + password-masked key + Save; read-only when env-managed); Devices pairing (create/join) is gated until the relay is configured. The key never ships (public fork, no baked default): unset ⇒ sync refuses to run against `.invalid`. Tests: resolveRelayEndpoint precedence (env/stored/ unconfigured), RelaySettingsStore round-trip + malformed/clear (main-db), transport sends X-Relay-Key from the resolved endpoint. node 3244, main-db sync 69, typecheck/oxlint/oxfmt clean. Devices/RelaySettings card UI is browser-suite coverage (not run in the agent gate).
…asks attachProject.mergeInto re-parented tasks/conversations/automations via raw SQL with no events, so a target project already mounted in the renderer did not show the merged-in tasks until an app restart (the common auto-attach case). Capture the re-parented task rows in the merge transaction and emit taskCreatedChannel for each under the target project id after commit, so a mounted TaskManagerStore ingests them live. +regression assertion in the merge test (with an @main/lib/events mock, since the real bus needs Electron).
Two pairing footguns from the review: - Revoke → permanent error loop: PairingService.getState only checks whether a local credential blob exists, so `paired` stayed true after a device was revoked, and SyncService treated the relay's 401 as a generic failure and looped in 'error' forever. SyncService now detects a permanent auth rejection (RelayHttpError 401/403) in the poll loop and calls a new onAuthRevoked dep (clears the credential + space key), dropping to idle so onboarding reappears. - Join-while-paired: PairingService.joinSpace unconditionally overwrote the credential/K0, so a mis-pasted secret silently switched spaces and dropped sync with the current devices. It now refuses with 'already_paired' when a credential for a *different* space exists (re-joining the same space is fine); the guard runs before any relay call. Tests: pairing already_paired guard (no relay call made); sync-service clears credentials + settles idle on a revoked poll (new 'revoked' fake mode).
…hild Applying a projects/tasks tombstone ran the raw SQLite ON DELETE CASCADE, which removed synced children (tasks/conversations/settings/remotes) outside the engine's per-row dirty guard — destroying a child that carried an unpushed local edit (spec #130 story 17: a pull must never clobber local edits). Before applying a parent delete, walk its synced descendants (the allowlist's FK graph) and, if any has an unpushed local edit, skip the delete (record the version as seen so it is neither re-fetched nor re-pushed); the next push flushes the child and a later delete applies cleanly once nothing is dirty. The row's own dirty tombstone is already handled by the existing per-row guard. Cascade-created child tombstones are intentionally still pushed, not cleared: the child may be alive on the relay (this machine's own upsert won an earlier LWW), so pushing its tombstone genuinely converges it, and a redundant push is an idempotent, harmless delete-of-a-delete. +regression test (dirty child survives a pulled parent delete).
apps/sync-relay was excluded from fork-ci, so its handlers (auth gate, LWW, counter, pairing, tombstone GC) had no CI coverage. Its tests run against an in-process node:sqlite D1 fake (no workerd), so add typecheck + oxlint + vitest steps for it to the gate.
Relay (apps/sync-relay): - Add authenticated POST /v1/space/delete, routed after the authenticate gate like the other authed endpoints. - Add deleteSpace(db, auth, now) in service.ts, backed by a new store.deleteSpaceRows() that wipes sync_rows, pull_cursors, tokens, join_secrets, version_counters, and the spaces row itself in one db.batch() call, matching createSpace's transactional pattern. - Tests: deleting a space wipes exactly that space's rows across every table and leaves a second space untouched; the endpoint 401s without a valid device token. - Document the endpoint in the README's protocol table plus a short "Deletion" section. Desktop client (apps/emdash-desktop): - RelayAuthApi + HttpRelayAuthApi: add deleteSpace(token), POSTing /v1/space/delete with the bearer token. - PairingService.deleteSpace(): calls the relay, then clears the local device credential and space key (un-pairs this machine) only after the relay confirms the delete, so a network/relay failure leaves the machine paired and free to retry. - Expose deleteSpace via the sync RPC controller; kicks the sync service so the status widget reflects "unpaired" immediately. - DevicesSettingsCard: add a destructive "Delete sync space" action behind the existing confirm-action modal, wired to rpc.sync.deleteSpace, refreshing state on success. - Tests: pairing.test.ts covers deleteSpace clearing the local credential and key after a successful relay call, leaving them intact on relay failure, and requiring a stored token.
The pasted pairing secret (`emdj1_<space>_<join>_<k0>`) had no integrity check: a mistyped or OCR'd character inside k0 or the join half still "parsed" successfully with silently wrong key material, surfacing only much later as an opaque decrypt failure (or a relay 401 for the join half) on the joining machine. Append a checksum segment covering the whole payload (space id bytes, join half, k0): `emdj1_<space 22>_<join b32 26>_<k0 b32 52>_<checksum b32 7>` (116 chars). The checksum reuses the device token's exact approach (truncated SHA-256, see checksumOf) but base32-encoded, since the rest of the secret already speaks base32. Relay (apps/sync-relay/src/crypto.ts): - composeSpaceSecret/makeSpaceSecret now compute and append the checksum; both become async (crypto.subtle.digest has no sync form), so service.ts's createSpace now awaits makeSpaceSecret. - Add CHECKSUM_B32_CHARS and grow SPACE_SECRET_CHARS. Client (apps/emdash-desktop/src/main/core/sync/crypto.ts): - composeSpaceSecret appends the checksum (node:crypto stays sync). - parseSpaceSecret recomputes the checksum from the parsed space id/join half/k0 and returns null on any mismatch, so joinSpace surfaces invalid_secret_format for a corrupted secret without calling the relay. - Add CHECKSUM_BYTES/CHECKSUM_B32_CHARS and grow SPACE_SECRET_CHARS. The join credential derivation (SHA-256 of the base32 join half, as stored by the relay) is unchanged; only the pasted secret gains a checksum. Tests: update every format regex/length assumption that assumed the old no-checksum layout (relay's crypto.test.ts + relay.test.ts, client's crypto.test.ts + pairing.test.ts + join-credential.test.ts). Add a cross-side test in join-credential.test.ts proving a relay-minted secret's checksum verifies on the client (and that a corrupted copy is rejected), plus flip-one-character tests in the client's crypto.test.ts and pairing.test.ts proving a typo anywhere in the payload is caught by parseSpaceSecret/joinSpace instead of surfacing later.
… replay Spec #130 amendment (anti-replay hardening), two parts: - Bind space_id into the AEAD AAD. crypto.ts's RowAad/aadOf now build [space_id, table, pk, version, keyId] instead of [table, pk, version, keyId], so a body encrypted for one space fails authentication if replayed into another. EncryptingRelayTransport takes the machine's own paired space id as a constructor argument (threaded from SyncCredential via SyncService) and binds it on both encrypt and decrypt — never the `space` field a pulled patch carries, which is exactly as untrusted as its `key_id` and would defeat the check if used to build the very AAD meant to authenticate it. Old envelopes need not decrypt (the relay is not deployed with data yet). - Guard against a relay replaying an old (but validly-encrypted) body under a fabricated newer server version: the existing serverVersion check alone accepts it, since server version is not part of the AAD. sync_row_state gains a client_version column (migration 0029) that tracks the client_version of the last genuinely pulled-and-applied patch per row. applyPatch now drops a pulled patch whose client_version regresses relative to the recorded one even though its server version is newer (counted in a new skippedReplayed field), while still advancing the recorded server version so the cursor never wedges. A NEVER_PULLED_CLIENT_VERSION sentinel keeps push-acks (which never touch client_version) from being confused with a genuine pulled baseline of 0 — otherwise two machines independently pushing the very same never-before-synced row (both legitimately client_version 0) would make each other's later edits look like replays of each other.
PairingService.deleteSpace() only cleared the local credential + space key on a relay success, so an 'unauthorized'/'device_not_found' response — the space was already deleted by another device, or a delete that committed server-side but whose response was lost — returned an error and left this machine paired. Retrying always 401s (the token's space is gone), so the explicit action could never un-pair, even though the background poll loop's onAuthRevoked already treats the same 401 as 'revoked, un-pair'. Treat unauthorized/device_not_found from the delete as 'already gone' and clear local state anyway; a genuine network/relay error still leaves the machine paired and free to retry. Adds the already-deleted test case. Review fix (spec #130 amendment, item: space deletion).
The single-character-flip test already proves the checksum catches substitutions; add an adjacent-transposition case (which preserves the character multiset, so only a position-sensitive checksum catches it) to lock in that guarantee too. Review nit (spec #130 amendment, item: pairing-secret checksum).
…ata loss Two spec #130 hardening changes to the E2E pull path. 1) Decrypt-failure quarantine + retry (new). Before, a pulled row whose body could not be decrypted was recorded as seen at its version and never retried — so a row encrypted under a space key this machine does not hold yet (a rekey whose new key has not propagated here) was lost forever, even after the key arrived. Now the encrypting transport tags each failure retryable (unknown key id / no key stored) or permanent (tampered/corrupt envelope, unsupported alg). Retryable failures are PARKED in a new sync_row_state.quarantined_version column (migration 0030, a plain ADD COLUMN — no rebuild) WITHOUT advancing server_version, and re-attempted by rewinding the pull cursor to the quarantine floor the next time the space key id changes (bounded to once per key change, never a per-cycle re-pull). A decrypted or deleted patch lifts the quarantine. Permanent failures keep the old drop-and-record-seen path. The standing quarantined count is surfaced in SyncStatus and the sidebar sync widget ('N rows can't be decrypted yet — waiting for the space key'). The engine stays crypto-free: it only receives the opaque key id. 2) Replay-guard data-loss fix (blocker, from review of the client_version anti-replay commit). The guard dropped a pulled patch whose client_version was <= the recorded baseline. But client_version is a last-observed server version, not a per-writer counter, so two machines that edited the same already-synced row both legitimately push the SAME client_version; the later one (higher server version) is a genuine concurrent edit LWW must apply, not a replay. '<=' silently lost it. Changed to '<': strict regressions are still caught (an equal-version replay of the identical body is a harmless idempotent re-apply). Adds a concurrent-tie regression test. Tests: migration 0030 (column added, client_version preserved, no rebuild), quarantine-then-retry-after-key-change, permanent-failure-not-retried, concurrent-edit-tie, updated the existing rekey/tamper undecryptable tests, and decryptRetryable classification assertions. Fixtures regenerated (pre-0030 frozen at 0029).
Merging fork-main brought in the external-content FTS5 workspace-file index (#147), whose `workspace_files_ad/ai/au` shadow triggers now co-exist with the sync clock/tombstone triggers. The 0025 sync-schema test asserted the FULL trigger set, so it broke on the merge. Filter to the `_sync_ts_` triggers this migration actually owns — matching the test's own 'exactly the portable tables' intent — so unrelated subsystems' triggers can't make it brittle.
CI (pinned Node 24) failed 'stores opaque row bodies verbatim': the test harness (node:sqlite) truncates a TEXT value at an embedded NUL byte (C-string semantics), so the '\u0000binary…' body came back as '' — a harness artefact, not relay behaviour. Node 26 (local) preserves it, which hid the failure. Swap the NUL-leading body for a NUL-free control-character body: the 'stored verbatim, never parsed as JSON' intent is preserved, and production bodies are always the E2E-encrypted base64url envelope (never a NUL byte), so no real coverage is lost.
The pairing client mapped every 404 from the relay to device_not_found
('That device was not found in this sync space'). The relay only returns
that error for a cross-space revoke; a bare 404 (unknown route, wrong
base URL, or a web page that is not the relay) was reported as if the
device had been deleted, sending users on a wild goose chase when their
relay URL was simply misconfigured.
Only map 404 to device_not_found when the relay says 'device not found
in this space'; everything else becomes relay_error with a message that
points at the relay URL in Settings.
The pairing client read space_id/device_id/device_token as spaceId/deviceId/deviceToken straight off the wire. The relay always answers snake_case, so createSpace received a valid 116-char secret but an undefined spaceId — parseSpaceSecret succeeded, the parts.spaceId !== spaceId guard fired, and every space creation failed with 'That doesn't look like a pairing secret'. Map the raw wire shapes to the camelCase DTOs at the API boundary (createSpace, joinSpace, listDevices) and add regression tests that assert the mapping with snake_case fixtures — the previous tests used a camelCase fake and never exercised the real wire format.
CI's tsgo flagged result.error.status on union-typed RelayApiError; add explicit type guards. Also record in AGENTS.md that every push must have its CI checked and any introduced failures fixed.
…s widget The first-run onboarding prompt and the status bar were two stacked sync surfaces that disagreed about the machine's state, and the relay not-configured case (no URL + key) was invisible outside Settings. Make SyncStatusWidget the single sync surface: - SyncStatus now carries relayConfigured/relayEnvManaged, resolved by SyncService each cycle from the injected endpoint resolver, so every status snapshot can say whether sync can run at all. - When the relay is unconfigured the widget row itself shows 'Sync isn't set up' and navigates to Settings -> Devices, where the relay form lives (covers paired machines whose relay was cleared too; clearRelaySettings kicks a status refresh). - The idle popover inherits the onboarding actions (Join an existing space / Start from scratch) and the SyncOnboardingPrompt is removed.
64ix
force-pushed
the
spec/130-multi-machine-sync
branch
from
August 11, 2026 09:21
c660341 to
424ba17
Compare
The sync engine writes pulled rows straight into the local DB, but the ProjectManagerStore loaded the project list once at boot and never re-read it, so projects synced from another machine stayed invisible for the whole session. Subscribe to the sync:status event and, on an up-to-date cycle, re-query and merge in any rows the boot load missed (spec #130 story 3: synced projects must appear without a restart). Merge-only: remotely deleted rows are reaped on the next launch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fork-only personal multi-machine sync for emdash: a self-operated Cloudflare Worker + D1 relay (apps/sync-relay) that orders encrypted, versioned row-bodies per space; the desktop app syncs an allowlist of Portable Data (projects minus local paths, task states, portable settings, prompt library, conversation metadata, automations configs) through a new main-process SyncEngine with injectable RelayTransport, last-write-wins by server-assigned versions, tombstones, and strict push-then-pull. Row bodies are AES-256-GCM encrypted end-to-end (per-row HKDF keys, AAD binding table/pk/client_version/key_id) with two-half pairing secrets (join_half + K0) so the relay never reads content; device-to-device pairing via copy-pasted secret, deep link emdash://join, and a Settings Devices screen. Synced projects arrive Unattached and are auto-attached by live-remote matching or attached/merged manually; automations arrive disabled with a source badge and resolve repository-instance targets at run time (workspace-config v3); a sidebar status widget shows syncing/up-to-date/offline-with-pending/error with an always-visible Sync now, plus a first-run Join-vs-start-from-scratch prompt and a not-resumable state for imported conversations.
Spec
Tickets Included
Closing References
Closes #130
Closes #131
Closes #132
Closes #133
Closes #134
Closes #135
Closes #136
Closes #137
Closes #138
Acceptance / Success Criteria
Modifications Par Intention
apps/sync-relay/src/index.tsapps/sync-relay/src/service.tsapps/sync-relay/src/store.tsapps/sync-relay/src/schema.tsapps/sync-relay/src/crypto.tsapps/sync-relay/src/db.tsapps/sync-relay/src/types.tsapps/sync-relay/test/apps/sync-relay/wrangler.jsoncapps/sync-relay/README.mdapps/sync-relay/package.jsonReview: Start with service.ts + store.ts (LWW, counter atomicity) and crypto.ts (secret/token formats) — the client must match these exactly.
apps/emdash-desktop/drizzle/0025_concerned_jetstream.sqlapps/emdash-desktop/drizzle/0026_nice_hex.sqlapps/emdash-desktop/drizzle/0027_old_junta.sqlapps/emdash-desktop/drizzle/0028_cloudy_nemesis.sqlapps/emdash-desktop/src/main/db/schema.tsapps/emdash-desktop/src/main/db/tests/migrations/apps/emdash-desktop/tooling/fixtures/Review: 0025 is the dangerous migration (cascade-free ordering); 0026 the trigger/clock design (echo-loop prevention); check the migration tests seed children.
apps/emdash-desktop/src/main/core/sync/engine.tsapps/emdash-desktop/src/main/core/sync/allowlist.tsapps/emdash-desktop/src/main/core/sync/row-state.tsapps/emdash-desktop/src/main/core/sync/transport.tsapps/emdash-desktop/src/main/core/sync/encrypting-transport.tsapps/emdash-desktop/src/main/core/sync/crypto.tsapps/emdash-desktop/src/main/core/sync/space-key-store.tsapps/emdash-desktop/src/main/core/sync/sync-secrets.tsapps/emdash-desktop/src/main/core/sync/sync-engine.db.test.tsReview: engine.ts dirty/echo-loop guards; allowlist.ts exclusions (machine-specific columns must never leak); crypto.ts AAD + key derivation; the no-plaintext-on-wire and two-client-K0 tests.
apps/emdash-desktop/src/main/core/sync/pairing.tsapps/emdash-desktop/src/main/core/sync/auth-api.tsapps/emdash-desktop/src/main/core/sync/sync-credentials.tsapps/emdash-desktop/src/main/core/sync/sync-controller.tsapps/emdash-desktop/src/main/core/sync/deep-link.tsapps/emdash-desktop/src/renderer/features/settings/components/DevicesSettingsCard.tsxapps/emdash-desktop/src/renderer/features/settings/components/join-sync-space-modal.tsxapps/emdash-desktop/src/renderer/features/settings/components/pairing-secret-modal.tsxapps/emdash-desktop/src/renderer/features/settings/sync-deep-link-handler.tsxReview: Secret handling (no logging, no persistence after use, no auto-join on deep link); the join-credential test drives the REAL relay service.
apps/emdash-desktop/src/main/core/projects/operations/attachProject.tsapps/emdash-desktop/src/main/core/projects/auto-attach.tsapps/emdash-desktop/src/main/core/projects/remote-matching.tsapps/emdash-desktop/src/main/core/projects/operations/getProjects.tsapps/emdash-desktop/src/renderer/features/projects/components/attach-project-modal.tsxapps/emdash-desktop/src/renderer/features/projects/components/UnattachedProjectPanel.tsxapps/emdash-desktop/src/shared/projects.tsReview: attachProject.ts merge/ambiguity logic; the engine import transforms (path strip/travel, collision nulling).
apps/emdash-desktop/src/main/core/sync/sync-service.tsapps/emdash-desktop/src/main/core/sync/sync-service-instance.tsapps/emdash-desktop/src/main/core/sync/sync-controller.tsapps/emdash-desktop/src/shared/core/sync/status.tsapps/emdash-desktop/src/shared/events/syncEvents.tsapps/emdash-desktop/src/renderer/features/sync/sync-store.tsapps/emdash-desktop/src/renderer/features/sync/sync-status-widget.tsxapps/emdash-desktop/src/renderer/features/sync/sync-onboarding-prompt.tsxapps/emdash-desktop/src/main/index.tsapps/emdash-desktop/src/main/shutdown.tsReview: sync-service.ts status transitions + backoff; the always-encrypted transport decision; widget browser tests.
apps/emdash-desktop/src/main/core/automations/apps/emdash-desktop/src/shared/core/automations/config.tsapps/emdash-desktop/src/shared/core/workspaces/workspace-config.tsapps/emdash-desktop/src/main/core/tasks/operations/createTask.tsapps/emdash-desktop/src/renderer/features/automations/apps/emdash-desktop/src/main/core/conversations/apps/emdash-desktop/src/renderer/features/conversations/Review: createTask.ts fail-before-task-create path; workspace-config v3 migrator + future-version preservation; the two-engine LWW test.
Review Guardrails
Per-unit reviews
Final integration review
Fixed after review
Points to double-check
pnpm exec oxlint .fails on fork-main itself (nested typeAware config, oxlint 1.69.0) — pre-existing, app-dir oxlint is greenQA Plan
Automated
Manual
Arbitrations / Decisions
Risks / Follow-ups