Storage reconcile: a failure between charge and watermark advance re-bills the interval

[2witstudios]

The window

reconcileSandboxStorage charges, then advances the row's watermark, as two separate un-transactioned writes:

chargeStorage(...)        // credit pipeline
advanceWatermark(now)     // a plain column UPDATE

If the process dies — or the UPDATE fails — between the two, the money moved but the watermark did not, so the next tick bills the same interval again. chargedButUnadvanced counts it; nothing prevents it.

Why it is like that

Deliberately, and the ordering is the safer half of a real trade:

• charge → advance (today): a crash before charging loses nothing (the window is retried), and a crash between re-bills one window.
• advance → charge: a crash between silently loses a window's revenue instead.

The module doc has said so since before drive environments existed. The window is also genuinely small — there is no I/O between the charge resolving and the UPDATE.

Why file it now

PR #2440 folds drive_envs into this meter as a second row source, which doubles the number of rows passing through that window, and spent its whole review cycle closing over-bill paths (a mid-tick watermark clobber, a stale provision timestamp, a frozen-span retroactive charge). This is the one over-bill path left, it is pre-existing, and fixing it properly is a schema change — so it belongs on its own rather than inside a dark-shipping billing PR.

Note this is not the same as the two paths #2440 closed. Those were arithmetic: a watermark moved to the wrong place. This one is a durability gap between two writes.

Options

1. Idempotency key on the charge, derived from (subjectKind, subjectId, billedThrough). A replay of the same interval is then a no-op at the ledger rather than a second debit. Needs a uniqueness constraint wherever charges land, and a decision about what the credit pipeline does with a duplicate.
2. A durable billed-interval record per subject — strictly more machinery than (1), but it also makes the meter auditable ("which intervals were billed, and when").
3. Flip the order to advance → charge, converting the rare over-bill into a rare under-bill. Cheapest by far, and consistent with the "under-billing is the accepted direction" principle the rest of this meter follows — but it reverses a documented deliberate choice, so it wants an explicit owner rather than a drive-by change.

(1) looks right if the credit pipeline can carry a key; (3) is the honest fallback if it cannot.

Acceptance

• A crash or failure between the charge and the watermark advance cannot produce a second debit for the same interval.
• chargedButUnadvanced either goes to zero or becomes purely informational.

Reported by CodeRabbit on #2440.