feat(serenity): dynamic AI resource allocation — consolidated allocator + kill-switch + hot-path wiring (LLMO-5616)

[byteclimber]

What & why

Just-in-time Semrush AI resource allocation — the parent-org pool is carved onto brand sub-workspaces up front (flat), which both under- and over-allocates on the tiny gold pool (13 projects / 800 prompts). This replaces the flat carve with a JIT top-up at the metering moment.

This PR is shaped directly by Rainer's direction on serenity-docs#22 (comment): consolidate the three approved-but-inert PRs into one and land them together with the actual handler-fronting, behind an OFF switch — "add" is not "enable."

Supersedes (will be closed once this is green)

This PR supersedes and replaces:

• feat(serenity): dynamic AI resource allocation — allocator core + transport read (LLMO-5616) #2749 — allocator core + transport read (brought in ~as-is; already reviewed).
• feat(serenity): LLMO/dynamic-allocation feature flag + auth threading (LLMO-5616) #2750 — per-org DB flag. Design changed (see below) — the per-org flag is dropped.
• feat(serenity): flag-gated carve — skip flat re-grant under dynamic allocation (LLMO-5616) #2751 — ensureSubworkspace carve-skip (brought in; now keyed off the global switch).

Key decisions (call-outs for reviewers)

1. Global kill-switch replaces the per-org DB flag (supersedes #2750)

Per Rainer's explicit direction, the per-org dynamic_allocation DB flag is replaced by a single global env/Vault boolean SERENITY_DYNAMIC_ALLOCATION (default OFF), read once per request off context.env — the same shape this repo already uses for its other global serenity toggles (SERENITY_ALLOW_WORKSPACE_DELETE, etc.). No PostgREST read, no per-org cache. Rollback is one env flip. Lives in dynamic-allocation-active.js.

2. Fail-fast hot path — NO settle poll on the synchronous request (behavior change vs the original plan)

A Semrush /resources transfer now gates every Adobe write. The original allocator blocked on a 12s settle poll + a 4-attempt not-ready retry (1+2+3s backoff) — that blows the ~15s Fastly/Lambda edge budget and risks a 504 with a half-applied absolute transfer stranded. Per Rainer's hard constraint, ensureAiHeadroom now does ONE transfer attempt, no settle poll; a still-settling child returns a retryable 503 (workspaceBusy, "provisioning, retry") immediately and the client/UI retries. The poll+retry transferAndSettle remains for the async/reconciler release path only (not the request path).

3. Actual handler-fronting (was never built), behind the OFF switch

Metered-write handlers front their metered op through a headroom guard:

• create-market: project headroom before createProject; prompt headroom before publishProject, sized from used + drafted (staleness-immune — the post-publish used reconciles async/stale-low).
• create-prompts (batch): one workspace-level prompt top-up (used + drafted) before publishAffected.
• update-models: publishedTexts × Δmodels before the shared syncModelsForProject publishes — fronted at the mode-guarded caller, not inside shared syncModelsForProject, so flat mode stays byte-for-byte.

4. Same-child concurrency lock

resource-lock.js serializes per-child top-ups (in-process), closing the stale-read-clobber-of-absolute-sets under-provision race for the intra-request mapLimit fan-out and same-container concurrency. Cross-container serialization is a PR-4 follow-up (see below).

5. Flag OFF is byte-for-byte

When OFF (or child/master ids missing), the guard is a genuine no-op — zero transport calls. Proven by the spy-based transport-call-sequence tests (dynamic-allocation-fronting.test.js) and by the full pre-existing suite passing unchanged with the flag defaulting OFF.

Enforcement choke point

dynamic-allocation-fronting.test.js asserts every subworkspace metered-write handler reads child headroom when the flag is ON — a future metered handler added without fronting fails this test.

Tests / gates

• npm test — 14167 passing, coverage 99.9% / 99.2% / 99.1% / 99.9% (gate: 90% aggregate).
• npm run lint clean; npm run type-check (blocking // @ts-check gate) clean.
• New/updated unit tests: fail-fast (single transfer, no poll, immediate 503), used+drafted publish sizing, need=0 no-op, partial/idempotent transfer, pool-exhaustion vs brand-limit vs transient-422, stale-low used, concurrency lock, flag-OFF byte-for-byte, enforcement choke point.

Explicitly deferred, tracked as follow-up (PR-4 rollout hardening)

Not built here (operational scope; the flag stays OFF until these land):

• Rightsizing sweep — one-time backfill of already-carved children (JIT only tops up; without it the pre-carve pool drain persists). Needs a background-job capability this path doesn't have today.
• Observability / SLIs — pool-free gauge, top-up p99, orgPoolExhausted/brandAiLimit rate, 405-classifier match ratio, not-ready-retry rate, release-failure rate + the paging split.
• Live-gateway canary — the disguised-405 body is still a substring guess; pin it via canary → fixture → contract test before flip-on.
• Cross-container serialization — the in-process lock covers intra-container; cross-instance needs a distributed lock / async worker.
• releaseAiSurplus wiring — built + tested but not wired into the delete request path (no async/reconciler mechanism exists yet; wiring it synchronously would block/strand). Consumer is the deferred reconciler.
• Zombie "not ready" recovery runbook, paging/alerting setup.

The §3 doc drift in serenity-docs (PROJECT_BLOCK 5→1) is fixed in a separate small serenity-docs PR.

Defaults

SERENITY_DYNAMIC_ALLOCATION defaults OFF — this PR adds capability, it does not enable it.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[MysticatBot]

Hey @byteclimber,

Verdict: Request changes - two material concerns around the advisory pool precheck producing spurious 409s and internal workspace IDs leaking in client-facing error messages.
Complexity: HIGH - large diff (2094 lines, 18 files); API surface signal.
Changes: Adds just-in-time Semrush AI resource allocation behind a global OFF-by-default kill-switch, replacing the flat pre-carve with a fail-fast headroom guard fronting all subworkspace metered-write handlers (18 files).

Must fix before merge

1. [Important] Advisory pool precheck throws spurious 409 when pool has capacity - src/support/serenity/resource-manager.js:258
2. [Important] Internal workspace IDs and pool numbers leak to clients via error messages - src/support/serenity/resource-manager.js:170

---

Finding 1: Advisory pool precheck is TOCTOU and can produce false 409s

The master pool free-space check (free = master.total - master.used) runs as a non-atomic advisory READ before transferOnce. Under concurrent requests (even across children of the same master), the precheck can see stale free space and throw orgPoolExhausted (409) when the transfer would actually succeed. The false-positive direction is the problem: the client gets a terminal 409 even though the pool had capacity when the transfer ran.

The code says "the transfer 422 is authoritative" - which is correct. But the advisory check fires a hard throw before the transfer gets a chance to be the authority.

Fix options:

• Demote from a hard throw to log.warn + proceed to transfer (let the 422 be the authority as intended)
• Remove the advisory precheck entirely (the transfer 422 handles the terminal case; the precheck only saves one round-trip in the exhausted case)
• Keep but document as fail-fast-conservative (may reject operations that would succeed under concurrent releases)

---

Finding 2: Error messages expose internal Semrush workspace UUIDs and pool numbers

orgPoolExhausted, brandAiLimit, and workspaceBusy embed internal details in ErrorWithStatusCode.message:

Org pool ${dim} free ${free} < needed ${delta} for ${childId}
Brand AI ${dim} allocation ${target} exceeds ceiling ${cap} for ${childId}
Sub-workspace ${workspaceId} is provisioning, retry

These flow through mapError to the client response. The existing redactUpstreamMessage pattern establishes that internal Semrush details should not reach API consumers.

Fix: Use a generic client message (e.g., "Resource pool exhausted") and log the detailed string with IDs/numbers separately at the throw site.

---

Non-blocking (6): minor issues and suggestions

• nit: handleUpdateModelsSubworkspace double-reads listSliceModels when allocation is ON (once for delta calc, once inside syncModelsForProject) adding an extra hot-path round-trip - src/support/serenity/handlers/markets-subworkspace.js:797
• nit: Controller lines ~1009 and ~1104 pass { dynamicAllocation: ... } without masterId, silently degrading to no-op guard when the flag is ON - add a comment confirming intentional - src/controllers/serenity.js:1009
• suggestion: isMeteredQuota matches bare substring quota which could misfire if upstream returns a legitimate 405 containing that word - match a more specific string - src/support/serenity/errors.js:86
• suggestion: resource-lock.js chains Map has no queue-depth cap - a stuck transport timeout blocks every subsequent caller for that child until Lambda times out - src/support/serenity/resource-lock.js:47
• suggestion: countPublishedPrompts lacks a dedicated unit test for the pagination-ceiling/truncation path (50-page cap, warn log, floor-count return) - src/support/serenity/handlers/markets.js:967
• suggestion: Serialization test in dynamic-allocation-active.test.js does not prove the second read sees post-transfer state (stub always returns zeros) - make the stub stateful to assert the critical section invariant end-to-end - test/support/serenity/dynamic-allocation-active.test.js:82

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 2m 30s | Cost: $9.88 | Commit: ee6077ffe6837189c302cfc095f9ddd1bcb61589
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[github-actions]

This PR will trigger a minor release when merged.

[MysticatBot]

Hey @byteclimber,

Verdict: Approve - both prior blocking findings are resolved; the new commit is clean and well-tested.
Complexity: HIGH - large diff (2094 lines, 19 files); API surface signal.
Changes: Adds just-in-time Semrush AI resource allocation behind a global OFF-by-default kill-switch, replacing the flat pre-carve with a fail-fast headroom guard fronting all subworkspace metered-write handlers (19 files).

Previously flagged, now resolved

• Advisory pool precheck is now genuinely advisory - logs a warning and proceeds to the transfer; the 422 "insufficient units" is the single authoritative exhaustion signal (no more spurious 409s from stale reads).
• Error messages are now generic with no internal workspace IDs or pool numbers - detailed context (childId, dim, free, delta) is logged separately at the throw site via structured log.warn calls.
• Serialization test is now stateful (proves the critical section serializes the read-then-absolute-set by asserting exactly one transfer when two concurrent ops each need +1 project).

Non-blocking (2): minor issues and suggestions

• nit: transferOnce added log parameter but the JSDoc @param {any} [log] is marked optional while transferAndSettle has it required in the signature - minor doc inconsistency - src/support/serenity/resource-manager.js:234
• suggestion: The countPublishedPrompts tests cover the page-ceiling and empty-project cases well; consider adding a test for when transport.listPromptsByTags rejects (error propagation on the pagination walk) - test/support/serenity/handlers/markets.test.js:1660

Note: CI build is still pending - verify it passes before merge.

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 5m 3s | Cost: $1.87 | Commit: 89f7a0df7ab7a4cfdc0e186d8727ce783a62353f
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[rainer-friederich]

Approving. Reviewed against the design (https://github.com/adobe/serenity-docs/pull/22) and the mock foundation (adobe/spacecat-shared#1767).

Architecture is sound and the implementation is faithful to the spec, with three deliberate, pre-approved deviations: the global env kill-switch (replacing the per-org DB flag), the fail-fast hot path (one transfer, no settle poll, retryable 503 on a still-settling child), and PROJECT_BLOCK = 1. The OFF path is proven byte-for-byte (spy-based transport-sequence tests + the full suite green with the flag defaulting OFF), so this is safe to merge and deploy to prod dormant — SERENITY_DYNAMIC_ALLOCATION defaults OFF and this changes nothing until the flag is flipped.

This lands the consume-side fronting only. The remaining work gates the flag flip, not this merge — all api-service fast-follows plus one ops step, tracked on the spec PR:

• fail-fast release wiring on delete / model-remove (inline, no new infra);
• the 405 guard — isMeteredQuota is built + unit-tested but not yet consumed anywhere in src;
• a per-brand ceiling — supported by ensureAiHeadroom but not passed by the controller today;
• an ON-path integration test (writable now against the mock __quota / PARENT_WITH_CHILD_METERED seed);
• a pre-flip guard asserting childId / masterId are present (a missing master currently silently no-ops the ON path);
• the live-gateway canary to pin the disguised-405 body and validate the post-transfer write race.

None of those block this merge.

[rainer-friederich]

Overall looks great, let's do the following:

1. rename masterId to parentWorkspaceId
2. rename childId to subWorkspaceId
3. Missing masterId | parentWorkspaceId should fail loud
4. Can we have integration tests against the "on" feature flag using our existing mocks? Needs a release of adobe/spacecat-shared#1767
5. Can we solve the open question regarding includeDrafted upfront?

This then needsw a follow up PR I guess with all the still required extnesions:

• ceiling threaded from the controller
• pre-flip guard.
• release-on-delete
• isMeteredQuota retry

Afterward we need acanary to do that.

[byteclimber]

Thanks @rainer-friederich — addressed in bb307b4cb:

1. ✅ masterId → parentWorkspaceId across the allocator, headroom guard, both fronting handlers, and the controller (aligns with the parentWorkspaceId vocabulary already used by workspace-resolver/brand-provisioning).

2. ✅ childId → subWorkspaceId everywhere.

3. ✅ Fail loud on missing id — new requireWorkspaceId guard in ensureAiHeadroom throws (→ 500) on a blank subWorkspaceId/parentWorkspaceId instead of silently reading ''. Unit test added.

   • Bonus the rename surfaced: handleCreateMarketSubworkspace was receiving the parent id both positionally (arg 3, for ensureSubworkspace) and redundantly in its options bag — dropped the redundant copy; the JIT pool is the positional parentWorkspaceId.

4. ✅ includeDrafted open question — resolved. Per design §2 (write-only metering: the prompt dimension is consumed at prompt write, not re-gated at publish), the sync's publish converting drafted prompts to used triggers no prompt-dimension quota gate, so sizing publishedTexts × netDelta is correct and includeDrafted is not needed at the update-models seam. (The includeDrafted on the create/bulk-prompt seams guards read-staleness of a subsequent op's used — a different concern.) Comment updated to state the decision; the §8 canary confirms §2 empirically. Please confirm §2 (write-only, no publish-time prompt gate) is the intended model — the resolution rests on it.

5. ⏳ IT against the flag ON — this is the one blocked item: it needs a published release of feat(user-manager-client): per-workspace AI resource accounting + /resources reads (LLMO-5616) spacecat-shared#1767 so we can bump the client dep and drive the mocks with dynamic_allocation ON. I'll add the flag-ON IT (extending test/it/postgres/serenity.test.js) as soon as feat: add impact field name #1767 is released. Tracking it so it isn't lost.

On the follow-up PR — agreed, I'll do the remaining extensions there as you scoped: ceiling threaded from the controller, the pre-flip guard, release-on-delete, and the isMeteredQuota retry, then the canary. I'll open it once this merges.

Green: type-check + lint clean; resource-manager.js and dynamic-allocation-active.js at 100% coverage; controller tests updated to assert the parent id is threaded positionally.

[MysticatBot]

Hey @byteclimber,

Verdict: Approve - rainer's feedback (rename, fail-loud guard, includeDrafted resolution) is addressed cleanly; no blocking concerns.
Complexity: HIGH - large diff (2471 lines, 21 files); API surface signal.
Changes: Renames masterId/childId to parentWorkspaceId/subWorkspaceId across the allocator, adds a fail-loud requireWorkspaceId guard, and resolves the includeDrafted design question per spec section 2 (21 files).
Note: CI build is still pending - verify it passes before merge.

Non-blocking (2): minor issues and suggestions

• nit: test/it/shared/tests/serenity.js and test/support/serenity/handlers/tags.test.js rename childId to subWorkspaceId in a tag-hierarchy context where the variable holds a child TAG id (a topic-category child of a root tag), not a sub-workspace id - a mechanical find-and-replace collateral that muddies the intent for the next reader. Consider reverting to childId or renaming to childTagId.
• suggestion: releaseAiSurplus in src/support/serenity/resource-manager.js does not call requireWorkspaceId like ensureAiHeadroom does - a blank subWorkspaceId would produce an opaque transport error rather than the clear 500. Adding the same guard would improve debuggability on the release path. - src/support/serenity/resource-manager.js:1046

Previously flagged, now resolved

• masterId renamed to parentWorkspaceId across the allocator, headroom guard, fronting handlers, and controller (vocabulary alignment with workspace-resolver/brand-provisioning).
• childId renamed to subWorkspaceId everywhere.
• Missing workspace id now fails loud via requireWorkspaceId guard (throws Error -> 500) instead of silently reading empty string.
• includeDrafted open question resolved per spec section 2 (write-only metering: no publish-time prompt gate).
• Serialization test is now stateful (proves the critical section serializes via exactly one transfer when two concurrent ops each need +1 project).

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 0m 4s | Cost: $4.45 | Commit: bb307b4cb7fd88053973dd5321435652624238c2
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[byteclimber]

✅ Item #4 (flag-ON integration test) — done in 7a068a401.

Now that adobe/spacecat-shared#1767 is merged + released (user-manager-client 1.5.0, with the metered mock image), I bumped the dep and added a flag-ON IT to the serenity suite:

• Serenity API — dynamic allocation ON (metered JIT via the live UM mock) — with the global SERENITY_DYNAMIC_ALLOCATION env ON and BRAND_1's sub-workspace seeded (via the mock's __quota control route) at a zero resource total, a real create-market fronts project headroom and tops the child up from 0 via a live /resources transfer to the mock. Asserting the total grew is a positive over-the-wire proof that the flag-ON JIT engaged (a flag-OFF run never transfers) — and it doubles as validation that the global kill-switch toggle takes effect per request.
• Gave ORG_1 a parent workspace so BRAND_1 resolves a non-null parentWorkspaceId (the guard's engage condition); added setUmMockQuota/dumpUmMock harness helpers.
• The prompt write runs as a smoke; the prompt texts × models sizing stays unit-test-covered (it's mock-metering nuance, not the IT's focus).

Verified locally against the live container stack: serenity IT 36 passing.

That closes all five of your immediate items (1–3 in bb307b4cb, #5 resolved there, #4 here). Remaining is the follow-up PR you scoped — ceiling threading, pre-flip guard, release-on-delete, isMeteredQuota retry, then the canary — which I'll open after this merges. Still flagging the separate pre-existing decommission {0,0} leak (out of this PR's diff) for its own ticket.

[MysticatBot]

Hey @byteclimber,

Verdict: Approve - clean flag-ON integration test; no blocking concerns.
Complexity: HIGH - large diff (2706 lines, 27 files); API surface signal.
Changes: Adds a flag-ON metered integration test for the JIT dynamic allocation path, proving the end-to-end top-up fires via a live /resources transfer to the UM mock (7 files).

Non-blocking (2): minor issues and suggestions

• nit: SERENITY_ORG_PARENT_WS_ID reuses the same UUID as SERENITY_MOCK_PROJECT_ID (b2c3d4e5-...) - sharing an identifier across semantically different entity types (project vs parent workspace) can make future test failures confusing to debug if both constants are referenced in the same test context. A distinct UUID would be clearer. - test/it/shared/seed-ids.js:76
• suggestion: childTotal(dump, dim) returns undefined on any shape mismatch (missing workspace_resources, missing CHILD record, missing ai sub-structure), which combined with .to.be.greaterThan(0) would produce a confusing assertion (expected undefined to be above 0). A guard assertion like expect(rec, 'child record not found in dump').to.exist before accessing the total would make failures actionable. - test/it/shared/tests/serenity.js

Previously flagged, now resolved

• Flag-ON integration test now covers the metered JIT top-up end-to-end via the live UM mock (Rainer's review item 4).
• @adobe/spacecat-shared-user-manager-client bumped to 1.5.0 (prerequisite for metered mock image).

Note: CI build is still pending - verify it passes before merge.

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 1m 5s | Cost: $4.35 | Commit: 7a068a4019aebdd86f8777ef22d3c34d09342e2c
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[rainer-friederich]

Re-reviewed the branch at 7a068a40. Items 1, 2 and 4 are done and verified; item 5 is settled (see below). Item 3 is not actually addressed — the guard that was added cannot be reached from the handler path. CI is green and this is still safe to merge dormant, but item 3 must not be recorded as closed.

Item 5 — confirmed. Spec section 2 (write-only metering: the prompt dimension is consumed at prompt write and is not re-gated at publish) is the intended model. The publishedTexts × netDelta sizing at the update-models seam is correct, and not passing includeDrafted there is right.

Item 3 — the fail-loud guard is unreachable in production

requireWorkspaceId is real and it does throw, but it sits at resource-manager.js:288, inside ensureAiHeadroom. createHeadroomGuard (dynamic-allocation-active.js:81) tests the ids first and, when either is missing, returns a no-op guard and never constructs the closure that calls ensureAiHeadroom:

const idsMissing = !subWorkspaceId || !parentWorkspaceId || ...
if (!enabled || idsMissing) {
  if (enabled) { log?.warn?.('... — skipping JIT top-up for this request', ...); }
  return { enabled: false, ensure: async () => ({ toppedUp: false }) };
}

ensureAiHeadroom has exactly one caller in src/ — that closure, which only exists when both ids are present. So requireWorkspaceId only ever fires from a unit test calling ensureAiHeadroom directly. On the request path the behaviour is unchanged: flag ON with a missing parentWorkspaceId is still a silent skip, now with a warn line. The code comment states the intent plainly — "a PR-4 pre-flip guard will assert the ids are present before the flag can be flipped on for real" — so this was deferred, not done.

Why it matters at flip time: under the flag the flat pre-carve is skipped as well, so a brand whose org has no parent workspace gets neither a carve nor a JIT top-up. Its sub-workspace sits at zero AI resources and the metered write fails at the Semrush gateway with an opaque error instead of the clear 500. The new IT is itself the evidence — ORG_1 had to be given a parent workspace for the guard to engage at all, which means the pre-existing seed orgs had none.

Please move the check into createHeadroomGuard (throw when enabled && idsMissing) in this PR rather than carrying it to PR-4. It is a few lines, and "flag ON but silently not metering" is exactly the failure mode a kill-switch rollout must not have. This also subsumes MysticatBot's first-round nit about the controller call sites degrading to a no-op guard when the flag is ON.

MysticatBot non-blocking findings still open

Worth doing before the flag flip:

• withResourceLock has no timeout on the predecessor promise (resource-lock.js:47) — raised twice, on 2026-07-06 and again on 2026-07-09. A hung transport call in a warm Lambda container blocks every subsequent write for that child until the container recycles. This one deserves a safety-valve timeout.
• releaseAiSurplus does not call requireWorkspaceId (resource-manager.js:374) — a blank subWorkspaceId produces an opaque transport error rather than the clear 500 that ensureAiHeadroom now raises.
• countPublishedPrompts propagates an unhandled rejection mid-walk (markets.js:967) — if listPromptsByTags throws partway through the pagination, the whole metered write fails. Wrapping it to return the count so far would match the truncation semantics already applied at the page ceiling. The suggested rejection-path test is also still missing.
• isMeteredQuota matches the bare substring quota (errors.js:97) — still text.includes('quota') || text.includes('allocation exhausted'). Acknowledged in the comment as pinned by the PR-4 canary, so this is deferred deliberately, but it is unchanged and a legitimate 405 containing the word would be absorbed.

Cosmetic, but should be cleaned up rather than dropped:

• Rename collateral in the tag tests — test/support/serenity/handlers/tags.test.js now has makeChildTreeTransport(rootId, subWorkspaceId) building a tag tree, and test/it/shared/tests/serenity.js renames a child tag id the same way. Those variables hold tag ids, not sub-workspace ids. childTagId would be right.
• releaseAiSurplus best-effort log interpolates values into the string (resource-manager.js:425) — `...for ${subWorkspaceId}: ${e?.message}` rather than the structured metadata form used elsewhere in the file.
• transferOnce JSDoc marks log optional (resource-manager.js:239) while transferAndSettle has it required (:193).
• SERENITY_ORG_PARENT_WS_ID reuses the exact UUID of SERENITY_MOCK_PROJECT_ID (test/it/shared/seed-ids.js:76) — b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e for two semantically different entities.
• childTotal(dump, dim) returns undefined on any shape mismatch (test/it/shared/tests/serenity.js), which with .to.be.greaterThan(0) yields "expected undefined to be above 0". An expect(rec).to.exist guard makes the failure actionable.

Deliberately deferred and fine as-is: the duplicate listSliceModels read on the flag-ON update-models path, and the advisory parent-pool read on every top-up — both are documented in-code with the PR-4 rationale.

Separate ticket

The pre-existing RELEASE_ALLOCATION = {ai:{projects:0,prompts:0}} leak at the four already-shipped call sites (workspace-lifecycle.js:43, most importantly decommissionBrandWorkspace) — please confirm the ticket exists before this merges, so it does not drop off the radar.

[rainer-friederich]

Follow-ups are now tracked

Everything deferred out of this PR — the rollout hardening listed in the description, the items I scoped for the follow-up PR, the open MysticatBot findings, and the pre-existing pool leak — now has a ticket. All four sit under epic LLMO-5005 and are assigned to @byteclimber.

LLMO-6190 — flag-flip readiness: correctness gates before SERENITY_DYNAMIC_ALLOCATION can be turned on

https://jira.corp.adobe.com/browse/LLMO-6190

The fail-loud guard for a missing parentWorkspaceId / subWorkspaceId (see my previous comment — the requireWorkspaceId check is currently unreachable from the request path); the per-brand ceiling threaded from the controller; releaseAiSurplus wired into market and brand delete; isMeteredQuota actually consumed so the disguised-405 becomes a top-up-and-retry; and the live-gateway canary to pin the 405 body as a fixture and contract test rather than a substring guess.

The fail-loud guard is still my ask for this PR. If it lands here, strike it from the story rather than doing it twice.

LLMO-6191 — rollout hardening: rightsizing sweep, observability, cross-container serialization

https://jira.corp.adobe.com/browse/LLMO-6191

The one-time rightsizing backfill of already-carved children (without it the pre-carve pool drain persists for every existing brand); pool-free gauge, top-up p99, exhaustion and classifier-match rates, not-ready retry rate, release-failure rate; the distributed lock for the absolute-set race across Lambda containers; the safety-valve timeout on withResourceLock, which MysticatBot raised twice and is unaddressed; and the zombie-workspace runbook plus paging.

LLMO-6192 — close out the open review findings

https://jira.corp.adobe.com/browse/LLMO-6192

releaseAiSurplus missing the requireWorkspaceId guard, and the countPublishedPrompts unhandled rejection mid-pagination — both worth doing before the flip. Then the hygiene set: the tag-test rename collateral (childTagId, not subWorkspaceId), the unstructured log line, the transferOnce JSDoc mismatch, the duplicated seed UUID, and the childTotal existence guard in the new IT. The duplicate listSliceModels read and the advisory parent-pool read are recorded there as deliberate, accepted trade-offs — no action.

LLMO-6189 — RELEASE_ALLOCATION zero-transfer is a silent no-op

https://jira.corp.adobe.com/browse/LLMO-6189

Filed as a Bug, since it is pre-existing on main, live today, and independent of this PR and of the flag. All four shipped call sites — decommissionBrandWorkspace, the ensureSubworkspace concurrency-loser release, and the two brand-provisioning.js failure-cleanup paths — send { ai: { projects: 0, prompts: 0 } } and log success while the gateway ignores it. Brand deactivation is therefore quietly shrinking the org's usable pool. It needs a design decision rather than a patch, because reclaim-to-zero is not expressible via transfer at all.

[byteclimber]

Thanks for the re-review @rainer-friederich — addressed in e299803b0:

Item 3 (BLOCKING) — fixed properly this time. You were right: requireWorkspaceId lived inside ensureAiHeadroom, which createHeadroomGuard only ever calls after deciding both ids are present — so the check never actually fired on the request path. Moved it into createHeadroomGuard itself: it now throws when enabled && idsMissing, instead of warn-and-degrade-to-no-op. Verified with a test that asserts the throw happens before any transport call, plus confirmed via grep that all three call sites (markets-subworkspace.js ×2, prompts-subworkspace.js) are bare statements in async functions with no wrapping try/catch, so the throw propagates cleanly to mapError's generic 500 branch.

MysticatBot's non-blocking findings — all done:

• withResourceLock safety-valve timeout (LOCK_TIMEOUT_MS, 10s default) — a hung predecessor no longer blocks the queue forever; the predecessor's own task isn't cancelled, only the queue stops waiting on it. Two new tests (hung-predecessor-unblocks / fast-predecessor-still-serializes).
• releaseAiSurplus now calls requireWorkspaceId too.
• countPublishedPrompts treats a mid-walk upstream failure as truncation (returns counted-so-far, same as the page-ceiling case) instead of propagating and failing the whole metered write. Added the missing rejection-path test.
• isMeteredQuota left untouched — per your own note, deliberately deferred to the PR-4 canary.

Cosmetic — all done:

• subWorkspaceId → childTagId in tags.test.js + the IT (these were tag ids, collateral from my earlier global rename sweep).
• releaseAiSurplus's best-effort log now uses structured metadata, not string interpolation.
• transferOnce/transferAndSettle JSDoc: log consistently optional.
• SERENITY_ORG_PARENT_WS_ID — I looked into making it genuinely distinct: the UM mock's exists() gate 403s any id that isn't a pre-registered workspace, the boot seed has exactly two, and the mock's internal seed-factories aren't reachable from api-service (external npm consumer, mock/* is monorepo-only). A third distinct workspace isn't practically available without a much bigger change (full store reseed, which risks other tests' baseline). Made the reuse an explicit, documented alias of SERENITY_MOCK_PROJECT_ID (single source of truth + rationale) rather than a silent duplicate literal.
• childTotal now guards expect(rec).to.exist before the numeric assertion, so a shape mismatch fails with an actionable message.

Separate ticket — confirmed: LLMO-6189 already exists and matches exactly (decommission/provisioning-cleanup RELEASE_ALLOCATION={0,0} no-op). Also found LLMO-6190/6191/6192 tracking flag-flip readiness / rollout hardening / this exact review's findings — this commit closes out LLMO-6192 in full and the createHeadroomGuard item + withResourceLock timeout item from LLMO-6190/6191 respectively.

Verified: type-check + lint clean, full unit suite 14303 passing, live serenity IT re-run against real containers — 36 passing (confirms the fix holds end-to-end, not just against mocks).

[MysticatBot]

Hey @byteclimber,

Verdict: Approve - clean fix commit addressing all of Rainer's review feedback; no blocking concerns.
Complexity: HIGH - large diff (2706 lines, 27 files); API surface signal.
Changes: Moves the fail-loud guard to the guard construction point (fixing reachability), adds a safety-valve timeout to the resource lock, and handles mid-walk upstream failures in prompt counting (12 files).

Non-blocking (1): minor issues and suggestions

• nit: The 12-line block comment explaining why SERENITY_ORG_PARENT_WS_ID aliases SERENITY_MOCK_PROJECT_ID is disproportionate to a single assignment - 3 lines ("aliases the mock project id because the UM mock 403s on unregistered workspace ids, and this is the only available mock workspace not already claimed by BRAND_1") would capture the essential reason without the implementation trivia about __seed and package.json exports - test/it/shared/seed-ids.js:72

Previously flagged, now resolved

• Fail-loud guard (requireWorkspaceId) moved from inside ensureAiHeadroom (unreachable from the handler path) to createHeadroomGuard construction - now actually fires when flag ON + ids missing (Rainer's item 3).
• Safety-valve timeout added to withResourceLock - a hung predecessor no longer blocks same-child ops forever (MysticatBot suggestion from prior reviews).
• countPublishedPrompts mid-walk upstream failure now returns counted-so-far instead of rejecting (MysticatBot suggestion).
• releaseAiSurplus structured logging fix (no more string-interpolated workspace ids).
• IT childTotal helper now has a guard assertion for actionable failure messages.
• Tag test variable renamed from subWorkspaceId to childTagId to eliminate vocabulary confusion with workspace allocator ids.

Note: CI build is still pending - verify it passes before merge.

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 0m 57s | Cost: $5.88 | Commit: e299803b010373ed7d9ac90aadd5b02081c46dac
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[rainer-friederich]

LGTM, please turn this on on prod and do the proper canary check with our testing org

[solaris007]

🎉 This PR is included in version 1.646.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀