Distinguish replication confirmation failure from a local write failure

DESIGN.md

@@ -72,6 +72,7 @@ Index of the design notes for the harper core: one line per note, grouped by the
 - [A local-only write marks both the record and its audit entry, and replay preserves it (`Table.ts` internal writes, `replayLogs.ts`)](resources/DESIGN.md#a-local-only-write-marks-both-the-record-and-its-audit-entry-and-replay-preserves-it-tablets-internal-writes-replaylogsts) — Mutations set `LOCAL_ONLY` on both persisted forms; messages set only their audit entry and preserve the row bit; crash replay reads the bit back from the entry (harper#2711).
 - [A record's residency is the listed nodes plus the writer, and a patch over a stub stays a stub (`Table.ts` `_writeUpdate`, harper#2257)](resources/DESIGN.md#a-records-residency-is-the-listed-nodes-plus-the-writer-and-a-patch-over-a-stub-stays-a-stub-tablets-_writeupdate-harper2257) — A local write never sheds the writer's complete copy; a patch onto an `INVALIDATED` stub is stored as a stub, never a complete row.
 - [`replicatedConfirmation` counts only the residents of each written record (`DatabaseTransaction` `confirmationTarget`, harper-pro#1031)](resources/DESIGN.md#replicatedconfirmation-counts-only-the-residents-of-each-written-record-databasetransaction-confirmationtarget-harper-pro1031) — The confirmation callback gets one other-resident list per distinct residency; a count any one list cannot meet fails before the commit.
+- [Replication confirmation failure means the linked writes committed locally](resources/DESIGN.md#replication-confirmation-failure-means-the-linked-writes-committed-locally) — Confirmation rejects only after local completions; its error carries `replicationUnconfirmed: true` in code and HTTP responses.
 - [This node's short id is 0, confirmed per audit store (`nodeIdMapping.ts` `getThisNodeId`)](resources/DESIGN.md#this-nodes-short-id-is-0-confirmed-per-audit-store-nodeidmappingts-getthisnodeid) — Audited writes skip re-reading the id-mapping record while the node name is unchanged; the per-worker confirmation expires so another worker's remap is picked up.
 
 ## resources/audit-retention.md — audit retention and cleanup

resources/DESIGN.md

@@ -978,16 +978,13 @@ Ids are resolved through a bounded per-database cache of frozen lists (`confirmi
 list. Count-based placement (`replicateTo: N`) keeps the writer out of the carried-over previous residency,
 or every update would repeat it and shed a replica.
 
-Every deterministic refusal precedes native submission, so the refused write never lands. A count larger
-than a record's list is refused in the commit handler before the write stages anything, and the refused
-write turns inert, so a caller that catches the refusal cannot commit it, even on a retry. Before native
-submission the commit rechecks the lists (confirmation may have been requested after a write was saved)
-and runs the feasibility check the replication layer registered with `replicationConfirmation` (harper-pro:
-the recipient bound). A table source's `before` hook (`source.put`/`patch`) has already run by then, because
-record-based placement needs the merged record; the guarantee covers Harper's stored data, not the source.
-An error after the native commit becomes a rejected completion, so the commit's bookkeeping still runs.
-LMDB is untouched: its commit never reaches a registered confirmation callback. Enforced by
-`unitTests/resources/residencyConfirmation.test.js`.
+Every deterministic refusal precedes native submission, so the refused write never lands. A count larger than a record's list is refused in the commit handler before the write stages anything, and the refused write turns inert, so a caller that catches the refusal cannot commit it, even on a retry. Before native submission the commit rechecks the lists (confirmation may have been requested after a write was saved) and runs the feasibility check the replication layer registered with `replicationConfirmation` (harper-pro: the recipient bound). A table source's `before` hook (`source.put`/`patch`) has already run by then, because record-based placement needs the merged record; the guarantee covers Harper's stored data, not the source. A confirmation-hook error after the native commit is observed after local completions, so bookkeeping and chained commits finish before a confirmation failure surfaces. LMDB's production registry is unregistered and its write-bearing child does not inherit the root's confirmation count. Feasibility and residency are pinned by `unitTests/resources/residencyConfirmation.test.js`; post-hook ordering is pinned by `unitTests/resources/replicationConfirmationOutcome.test.js`.
+
+## Replication confirmation failure means the linked writes committed locally
+
+`ReplicationConfirmationError.replicationUnconfirmed` is true only for a post-commit confirmation-hook failure. `confirmReplicationAfterCommit` observes the hook immediately, and `settleCommitCompletions` waits for local work first (a child's own confirmation failure, requiring an explicitly assigned link count, is deferred too). Local write/flush failures reject promptly without the flag. The wrapper retains the original public code, message and status through `cause`, without mutating a shared rejection, and advertises `retryable: false`: replay could repeat committed writes. REST problem details, operations error bodies and terminal stream records expose the flag. This is local commit, the same durability as success, not a peer or fsync guarantee. A failed commit ends its scope; it cannot resume writes. Only a transaction's root link inherits its context's count; LMDB's write-bearing child and later database links are not automatically confirmed. Pinned by `unitTests/resources/replicationConfirmationOutcome.test.js` and `unitTests/server/serverHelpers/serverErrorHandler.test.js`. A confirmation error from an independent nested transaction cannot attest to an enclosing scope's pending writes: the commit and callback failure funnels remove its flag before aborting those writes. Abort and terminal-failure cleanup latch discarded writes on the logical scope's root, so an earlier timeout, disconnect or explicit abort cannot erase that evidence. Committed or still-submitted writes and reused self-committing instances do not set the latch. The funnels keep the original error in `cause` and `retryable: false` in programmatic and HTTP errors because the nested write did land. The flag attests to the reported commit's issued writes, not later application steps that never ran after a rejection; a nested flag can remain when the enclosing scope has neither pending nor previously discarded writes.
+
+Programmatic errors are `ReplicationConfirmationError` wrappers, not the original instance; inspect `cause` for its class and identity. String-valued errors (native HTTP handlers, WebSocket close reasons and JSON-array stream elements) render `ReplicationConfirmationError`. Object responses preserve the original public code alongside the flag. Confirmation wrapping and flag removal copy diagnostics best-effort, preserving the cause and outcome when getters fail. Unreadable structured confirmation details fall back to the ordinary error response. HTTP guards cover the `replicationUnconfirmed`/`retryable` extensions and structured confirmation details. REST and operations renderers' base reads of `statusCode`, `code`, `message` and `detail` are not hardened for arbitrary application errors.
 
 ## This node's short id is 0, confirmed per audit store (`nodeIdMapping.ts` `getThisNodeId`)
 

resources/DatabaseTransaction.ts

@@ -5,6 +5,8 @@ import {
 	ClientError,
 	DatabaseClosingError,
 	DatabaseDrainTimeoutError,
+	ReplicationConfirmationError,
+	withoutReplicationUnconfirmed,
 	ServerError,
 	TransactionCommitConflictTimeoutError,
 } from '../utility/errors/hdbError.ts';
@@ -487,6 +489,39 @@ export function replicationConfirmation(callback, feasibility?: typeof confirmat
 	confirmReplication = callback;
 	confirmationFeasibility = feasibility;
 }
+
+export function confirmReplicationAfterCommit(callback: () => unknown): Promise<void> {
+	let result;
+	try {
+		result = callback();
+	} catch (error) {
+		result = Promise.reject(error);
+	}
+	const confirmation = Promise.resolve(result).then(
+		() => {},
+		(error) => {
+			throw new ReplicationConfirmationError(error);
+		}
+	);
+	confirmation.catch(() => {});
+	return confirmation;
+}
+
+export function settleCommitCompletions(completions: unknown[], confirmation?: Promise<void>): Promise<void> {
+	if (completions.length === 0) return confirmation ?? Promise.resolve();
+	let deferredConfirmationError: ReplicationConfirmationError;
+	const localCompletions = completions.map((completion) =>
+		Promise.resolve(completion).catch((error) => {
+			if (error instanceof ReplicationConfirmationError) deferredConfirmationError ??= error;
+			else throw error;
+		})
+	);
+	// A confirmation failure must not abort a chained local commit that has yet to land.
+	return Promise.all(localCompletions).then(() => {
+		if (deferredConfirmationError) throw deferredConfirmationError;
+		return confirmation;
+	});
+}
 let txnExpiration = envMngr.get(CONFIG_PARAMS.STORAGE_MAXTRANSACTIONOPENTIME) ?? 30000;
 
 class StartedTransaction extends Error {}
@@ -1000,6 +1035,8 @@ export class DatabaseTransaction implements Transaction {
 	declare scopeAbandoned?: boolean;
 	declare abandonedLinks?: DatabaseTransaction[];
 	committingWrites = false;
+	// Root-scoped history survives abort cleanup until the logical scope ends.
+	declare writesRolledBack?: boolean;
 
 	rangeReadActive = false;
 
@@ -1631,6 +1668,25 @@ export class DatabaseTransaction implements Transaction {
 		return false;
 	}
 
+	private hasWriteToDiscard(
+		nativeTxn: RocksTransaction | null | undefined,
+		targetTransaction: RocksTransaction | undefined,
+		hasSiblingInFlight: boolean,
+		preserveActiveWrites: boolean
+	): boolean {
+		for (let i = 0; i < this.writes.length; i++) {
+			const write = this.writes[i];
+			if (
+				!write ||
+				(preserveActiveWrites && this.isWriteOwnedByActiveCommit(write, targetTransaction, hasSiblingInFlight, true))
+			)
+				continue;
+			const owner = write.stagedOn ?? this.activeCommitHandle ?? nativeTxn;
+			if (!this.isHandleCommitted(owner) && !hasDurableLog(owner)) return true;
+		}
+		return false;
+	}
+
 	private hasSavedWriteForHandle(transaction: RocksTransaction, isImmediateCommit: boolean): boolean {
 		for (let i = 0; i < this.writes.length; i++) {
 			const write = this.writes[i];
@@ -2964,6 +3020,7 @@ export class DatabaseTransaction implements Transaction {
 								const store = this.writes[0]?.store;
 								if (store) completions.push(store.flushed);
 							}
+							let confirmation: Promise<void>;
 							if (this.replicatedConfirmation) {
 								// if we want to wait for replication confirmation, we need to track the transaction times
 								// and when replication notifications come in, we count the number of confirms until we reach the desired number
@@ -2975,7 +3032,7 @@ export class DatabaseTransaction implements Transaction {
 											? this.confirmationTarget(committedLogKey, committedTransaction)
 											: undefined;
 									if (target) {
-										completions.push(
+										confirmation = confirmReplicationAfterCommit(() =>
 											confirmReplication(
 												databaseName,
 												target.version,
@@ -2986,7 +3043,7 @@ export class DatabaseTransaction implements Transaction {
 									}
 								} catch (error) {
 									const refusal = Promise.reject(error);
-									refusal.catch(() => {}); // surfaces through Promise.all(completions), even if bookkeeping throws first
+									refusal.catch(() => {}); // surfaces through local completions, even if bookkeeping throws first
 									completions.push(refusal);
 								}
 							}
@@ -3021,7 +3078,11 @@ export class DatabaseTransaction implements Transaction {
 								!this.hasActiveCommitHandle()
 							)
 								this.timestamp = 0; // reset the timestamp as well
-							return Promise.all(completions).then(
+							const completed =
+								confirmation || (this.root ?? this).replicatedConfirmation
+									? settleCommitCompletions(completions, confirmation)
+									: Promise.all(completions);
+							return completed.then(
 								() => {
 									// Only once the chained store's commit has settled, as on the synchronous path: a
 									// partially failed mid-scope commit must not leave the scope resumable.
@@ -3146,6 +3207,12 @@ export class DatabaseTransaction implements Transaction {
 									this.releaseContext(!!options.doneWriting);
 								}
 								if (!hasDurableLog(transaction, error as any)) {
+									if (
+										!isImmediateCommit &&
+										!this.saveCommits &&
+										this.writes.some((write) => write && (!write.stagedOn || write.stagedOn === transaction))
+									)
+										(this.root ?? this).writesRolledBack = true;
 									for (const write of this.writes) {
 										if (
 											!write?.saved ||
@@ -3339,6 +3406,10 @@ export class DatabaseTransaction implements Transaction {
 	}
 
 	protected abortAfterCommitError(error, targetTransaction?: RocksTransaction): never {
+		try {
+			if (error?.replicationUnconfirmed === true && (this.hasPendingWrites() || (this.root ?? this).writesRolledBack))
+				error = withoutReplicationUnconfirmed(error);
+		} catch {}
 		try {
 			const retryTxn = targetTransaction as RocksTransactionWithRetry | undefined;
 			const retryTxnCommitted = this.isHandleCommitted(retryTxn);
@@ -3371,6 +3442,13 @@ export class DatabaseTransaction implements Transaction {
 			isImmediateTarget || this.hasWriteOwnedByActiveCommit(targetTransaction, hasSiblingInFlight, true);
 		const preserveNonImmediateActiveWrites =
 			isImmediateTarget || this.hasWriteOwnedByActiveCommit(targetTransaction, hasSiblingInFlight, true, true);
+		if (
+			this.writes.length > 0 &&
+			!isImmediateTarget &&
+			!this.saveCommits &&
+			this.hasWriteToDiscard(nativeTxn, targetTransaction, hasSiblingInFlight, preserveActiveWrites)
+		)
+			(this.root ?? this).writesRolledBack = true;
 		const cascadeNext = cascade && !preserveNonImmediateActiveWrites;
 		const next = cascadeNext ? this.next : undefined;
 		if (cascadeNext && !retainReadTransaction) this.next = null;

resources/LMDBTransaction.ts

@@ -7,12 +7,14 @@ import {
 	shouldSpareCommitPhase,
 	requestAbortedError,
 	transactionOpenTooLongError,
+	confirmReplicationAfterCommit,
+	settleCommitCompletions,
 	type CommitOptions,
 	type TransactionWrite,
 	type CommitResolution,
 } from './DatabaseTransaction';
 import { cleanupUnusedBlobs, collectRetainedFileIds } from './blob.ts';
-import { ServerError } from '../utility/errors/hdbError.ts';
+import { ServerError, withoutReplicationUnconfirmed } from '../utility/errors/hdbError.ts';
 import { getNextMonotonicTime } from '../utility/lmdb/commonUtility.ts';
 import * as harperLogger from '../utility/logging/harper_logger.ts';
 import type { Context } from './ResourceInterface.ts';
@@ -336,13 +338,14 @@ export class LMDBTransaction extends DatabaseTransaction {
 						const store = this.writes[0]?.store;
 						if (store) completions.push(store.flushed);
 					}
+					let confirmation: Promise<void>;
 					if (this.replicatedConfirmation) {
 						// if we want to wait for replication confirmation, we need to track the transaction times
 						// and when replication notifications come in, we count the number of confirms until we reach the desired number
 						const databaseName = this.writes[0]?.store.rootStore.databaseName;
 						const lastEntry = this.lastConfirmableEntry();
 						if (confirmReplication && databaseName && lastEntry)
-							completions.push(
+							confirmation = confirmReplicationAfterCommit(() =>
 								confirmReplication(databaseName, (lastEntry as any).localTime, this.replicatedConfirmation)
 							);
 					}
@@ -359,7 +362,11 @@ export class LMDBTransaction extends DatabaseTransaction {
 					this.clearWrites();
 					this.timestamp = 0;
 					this.next = null;
-					return Promise.all(completions).then(() => {
+					const completed =
+						confirmation || (this.root ?? this).replicatedConfirmation
+							? settleCommitCompletions(completions, confirmation)
+							: Promise.all(completions);
+					return completed.then(() => {
 						return {
 							txnTime,
 						};
@@ -394,6 +401,11 @@ export class LMDBTransaction extends DatabaseTransaction {
 	}
 
 	private abortCommitFailure(error, next: DatabaseTransaction | undefined): never {
+		try {
+			if (error?.replicationUnconfirmed === true && (this.hasPendingWrites() || (this.root ?? this).writesRolledBack))
+				error = withoutReplicationUnconfirmed(error);
+		} catch {}
+		if (!this.saveCommits && this.writes.some((write) => write)) (this.root ?? this).writesRolledBack = true;
 		const nextWasAttached = Boolean(next && this.next === next);
 		try {
 			this.abort(true);
@@ -422,6 +434,13 @@ export class LMDBTransaction extends DatabaseTransaction {
 	}
 
 	abort(retainReadTransaction = false, cascade = true): void {
+		if (
+			this.writes.length > 0 &&
+			!this.saveCommits &&
+			!this.nativeCommitSubmitted &&
+			this.writes.some((write) => write)
+		)
+			(this.root ?? this).writesRolledBack = true;
 		const next = cascade ? this.next : undefined;
 		if (cascade && !retainReadTransaction) this.next = null;
 		try {

resources/transaction.ts

@@ -8,6 +8,7 @@ import {
 } from './DatabaseTransaction.ts';
 import { AsyncLocalStorage } from 'async_hooks';
 import * as harperLogger from '../utility/logging/harper_logger.ts';
+import { withoutReplicationUnconfirmed } from '../utility/errors/hdbError.ts';
 
 export const contextStorage = new AsyncLocalStorage<Context>();
 
@@ -104,6 +105,14 @@ export function transaction<T>(
 		abortAndThrow(error, true);
 	}
 	function abortAndThrow(error, callbackThrew: boolean): never {
+		try {
+			if (
+				callbackThrew &&
+				error?.replicationUnconfirmed === true &&
+				(transaction.hasPendingWrites() || transaction.isChainCommitting() || transaction.writesRolledBack)
+			)
+				error = withoutReplicationUnconfirmed(error);
+		} catch {}
 		// A commit attempt that has not reached its native outcome owns its own teardown — a handler that
 		// fired txn.commit() without awaiting it can get here while it is still running, and aborting
 		// would clear the writes it is committing and abort the handle it is committing them through.

server/DESIGN.md

@@ -226,7 +226,13 @@ SSE and NDJSON serializers eagerly take and hold their first iterator step for G
 
 SSE terminal event data uses `{ error: <code-or-class>, message: <message>, status?: <status> }`; NDJSON wraps the same object as `{ "$harperStreamError": { ... } }` so it cannot be mistaken for an ordinary row with an `error` field.
 Sources with `mapError` remain unmapped for these two formats so their failures reach this single contract instead of becoming legacy data records.
-generic JSON-array streaming retains its older `{ error: "<name>: <message>" }` element shape.
+Generic JSON-array streaming retains its `{ error: "<name>: <message>" }` element shape.
+REST problem details publish only an explicitly negative `retryable: false` hint; a missing hint is unknown.
+All three terminal records add `replicationUnconfirmed: true` for a reported commit whose local writes landed
+but whose confirmation failed, and `retryable: false` for that outcome or another explicitly non-retryable error.
+An enclosing scope that rolls back, or has already discarded, writes removes the confirmation flag while preserving the negative retry hint. Metadata getters
+on a thrown value may fail; omit unreadable extensions and still finish a valid error record. The commit-level
+scope and producer coverage are described in [resources/DESIGN.md](../resources/DESIGN.md#replication-confirmation-failure-means-the-linked-writes-committed-locally).
 The `error` value is the stable programmatic discriminator; `message` is diagnostic and follows the same thrown-message exposure policy as pre-commit Problem Details.
 
 Clean stream completion does not prove completeness; clients must inspect SSE for the `harper-error` event and NDJSON for the `$harperStreamError` control record.