Drive envs: a failed provision-time storage measurement is never retried

[2witstudios]

The gap

A drive environment has exactly one storage-measurement writer: the provision-time baseline wired at envStorageMeasureSeam (packages/lib/src/services/drive-envs/env-storage-measure.ts), fired by ensureSpriteHolderSandbox on the create arm only.

That single write is not retried. If it fails — an exec throw, a du timeout, unparseable output, or a recordStorageMeasurement rejection — drive_envs.storageMeasuredBytes stays NULL, and nothing ever writes it again.

The storage reconcile then does exactly what it is designed to do, which makes the loss permanent rather than deferred: it bills the never-measured 0 floor and advances the watermark anyway (sandbox-storage-reconcile.ts, the costDollars <= 0 branch). Advancing unconditionally is deliberate and correct — freezing the watermark would let a later measurement retroactively over-bill the entire frozen span — but it means each skipped window is discarded for good. Once the warm path lands, there is no backfill: the env simply starts billing correctly from that point on, and everything before it was free.

Same shape, worse consequence, for a rebuild: revivedDriveEnvColumns re-stamps storageLastBilledAt but leaves storageMeasuredBytes at the previous generation's value. The baseline measurement on the fresh (empty) disk is what corrects it; if that write fails, the env keeps billing the dead generation's footprint — an over-bill, on a real payer, indefinitely.

Sessions have the same single-write hole in principle, but three writers in practice (provision, the bash path's post-op measureStorage, the git path's), so a dropped measurement self-corrects on the next real work. An env has none of those until env-bound sessions exist.

Why it wasn't fixed in #2440

The natural retry point is a re-measure on the resume arm of ensureSpriteHolderSandbox — but that is the holder-neutral provisioning core shared with sessions, so changing it there would alter session provisioning behaviour inside a PR whose whole premise is that folding envs in changes nothing about the existing meter. #2441 (pu/env-sessions) owns the env ensure path and is the natural home.

What #2440 did instead

Made the failure observable rather than silent:

• every failure path in the seam now logs with the env's id;
• the reconcile counts neverMeasured — live rows with no reading at all;
• measurementHealth splits that per persistence unit, so an env-side hole is visible separately from a session-side one.

A metric is not a fix, which is why this issue exists.

Options

1. Re-measure on resume when the row has no reading. Cheapest and self-healing: fire the seam on the resume arm when storageMeasuredBytes IS NULL, so the next person to open the env repairs it. Needs a decision about whether sessions get the same treatment.
2. Warm refresh from env sessions (the planned path). Fixes the steady state and incidentally repairs a failed baseline on first real work. Does not help an env nobody opens.
3. A backfill sweep for rows that have been neverMeasured across N ticks. Most thorough, most machinery, and it must still never wake a paused Sprite — so it can only measure envs that happen to be awake.

Option 1 plus 2 probably covers it. Option 3 only if neverMeasured proves non-transient in practice.

Acceptance

• A provision-time measurement failure is repaired without human intervention.
• A rebuilt env cannot keep billing the previous generation's footprint.
• neverMeasured returns to ~0 in steady state, so it stays usable as a signal.

[2witstudios]

Correction

The original text of this issue said:

Same shape, worse consequence, for a rebuild: revivedDriveEnvColumns re-stamps storageLastBilledAt but leaves storageMeasuredBytes at the previous generation's value. […] if that write fails, the env keeps billing the dead generation's footprint — an over-bill, on a real payer, indefinitely.

That is wrong. reviveStamps (plan-workspace-lifecycle.ts) sets storageMeasuredBytes: null and storageMeasuredAt: null, and revivedDriveEnvColumns applies them via envStampColumns. The columns are cleared on the create arm.

The real post-rebuild outcome of a failed measurement is a NULL row billing the 0 floor — an under-bill, the same direction as every other case here, never an over-bill. Nothing in this issue's scope changes; the severity is simply lower than stated, and the wrong claim is corrected in the module docs too (PR #2440, edb6003be).

One case is now fixed rather than deferred

The adopt arm — reached when the platform replaces a VM under the holder's deterministic name — also nulls the measurement columns, and it was not firing measureStorage at all. For a session that self-heals via the bash/git writers; for an env it was permanent. PR #2440 now fires the seam on both arms that clear the measurement (create and adopt) and on neither of the two that don't.

What remains for this issue

A failed measurement on either arm is still unretried, and because the reconcile advances the watermark for a 0-floor row anyway, each skipped window is discarded rather than deferred. The options in the original description stand — re-measure on resume when the row has no reading, warm refresh from env sessions, or a backfill sweep — with option 2 now the most natural home, since adopt no longer needs its own repair.

[2witstudios]

Adding to this issue's scope, from review of #2440.

The adopt arm clears the measurement and cannot repair it

planSpriteHolderLifecycle's adopt verdict — reached when the platform has replaced a holder's VM under its deterministic Sprite name — nulls storageMeasuredBytes and storageMeasuredAt in the same write that records the new identity. Correct: a replacement is a different disk, so the old reading describes storage that no longer exists.

Nothing then re-measures. #2440 briefly did fire measureStorage on that arm, and reverted it, because doing so can wake a hibernating Sprite:

• a du is an exec, and an exec is the only way to wake a hibernated Sprite — there is no wake API (stream()'s docblock in sprite-sandbox-host.ts);
• the create arm may measure because host.provision has just booted the VM;
• adopt's only evidence the Sprite exists is probeRecordedSprite → host.attach, which is a control-plane client.get and says nothing about whether the machine is running.

So measuring there could resume a paused VM and restart its runtime billing — the Phase-3 keep-awake bug that sandbox-storage-reconcile.ts forbids outright. Under-billing that row is the accepted direction in this meter; over-billing a machine nobody asked to run is not.

Net effect: an adopted holder sits at the never-measured 0 floor until something else measures it. For a session, the bash/git writers repair it on the next real work. For an env there is no such path today, so it is permanent — the same shape as the failed-baseline case this issue already covers.

What would close it

Any of these, in rough order of appeal:

1. Measure where the VM is known awake. The warm path (a session's tool run, an env session's first real work) already holds a running sandbox. This is option 2 in the original description and now covers the adopt case as well.
2. Give the probe a running-state signal. If host.attach/the SDK can report whether the machine is running without waking it, the adopt arm could measure only when it is — and this becomes a two-line change rather than a policy question.
3. Leave it, and rely on (1) alone once env sessions exist.

Nothing here changes the acceptance criteria already listed; it adds one more way a holder reaches the 0 floor with no writer to rescue it.

[2witstudios]

One more route into this gap, from a later review pass on #2440.

The env's baseline du races an HTTP response

envStorageMeasureSeam is wired at buildEnvProvisionDeps in apps/web/src/lib/drive-envs/drive-envs-runtime.ts, and rebuildEnv — a Next.js route handler — is the only env provisioning path today. The measurement is fire-and-forget by contract (a billing observation must never delay a provision) and the du is capped at 20s, so it routinely outlives the response.

A container drain or deploy landing in that window kills the exec. The row keeps its NULL baseline, and since this is an env's only measurement writer, nothing retries it.

The session equivalent has no such exposure: its provisioning hook runs inside apps/realtime, a long-lived process.

So the ways an env reaches a permanent 0 floor are now three, all with the same fix-shape:

1. the baseline du fails (exec throw, timeout, unparseable output, or a persist that rejects);
2. the adopt arm clears the measurement and deliberately does not re-measure (it cannot prove its VM is awake);
3. the baseline du is killed mid-flight by the request context going away.

None of them are repaired by anything today, and because the reconcile advances the watermark for a 0-floor row, each skipped window is discarded rather than deferred.

All three are closed by the same thing: a measurement taken somewhere that already holds a running sandbox and is not racing a response — the warm path, i.e. a session doing real work inside the env. That remains option 1/2 from the original description; this just adds a third reason it is the right home rather than a nice-to-have.