Skip to content

Verify complete static-sharding reads and pin local-only filtering fix - #1130

Merged
kriszyp merged 5 commits into
mainfrom
fix/static-sharding-limit-regression
Oct 27, 2026
Merged

kriszyp merged 5 commits into
mainfrom
fix/static-sharding-limit-regression

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 26, 2026 •

Copy link
Copy Markdown
Member
An engineer inspects three returned records

⊙ Problem

The static-sharding test still expects an empty pooled nonowner query after #1087 made invalidation stubs searchable and ordinary queries hydrate their full records. Strengthening it exposed a runtime defect: local-only queries returned partial {id, tag} stubs. The pooled PR job also omitted this suite.

🤔 Keep complete results and placement guarantees

Keep #1087: pooled reads return three distinct complete matches; unpooled and local-only reads return []. Both nonowners retain their stub flags, payloads and residency.

💡 Solution

Pin the coordinated core fix and retain the strengthened static-sharding assertions. Core's Table.transformEntryForSelect now checks invalidation flags after lazy primary-entry loading. Core #3527's rewrite must preserve this filter and the local-only SKIP handling in Table.transformToOrderedSelect, plus the null-safe field access in getAttributeValue and the Table.search flag enabling the collection filter before sort-key resolution.

The fixture observes raw metadata and waits for returned rows' replica-storage attempts to settle. The existing pooled PR job now includes this suite. The design note describes indexed-stub hydration and its cross-shard limit.

⚖️ Alternatives

🤔 Preserve the local-only assertion

Returning partial stubs is the defect. Weakening the assertion or changing invalidation indexing is unsuitable. Cross-shard merged limits remain #198/#436. This test checks complete results from existing local candidates, not a shard-wide search or speculative-fetch budget.

🤔 Keep the current PR runtime selection

Main/nightly already run Node 22/24/26. Add this suite to the pooled PR job without tripling its five unrelated suites; validate all twelve Node/pool/engine combinations locally. The fixture's completion fence follows the current admission call and must change if that call is removed.

Planning decision and CI capacity

Framing-Verdict: better-alternative-exists, resolved by adopting the narrower CI change and deterministic completion fence, then coordinating the required runtime repair. The updated pooled PR job passed in 4m02s, including setup and this suite (43 passing, two skips). Retain the 25-minute cap.

✅ Verification

The final twelve-leg matrix ran on Pro1201ab2b/corec1e192c7: 11/12 legs pass. Node24/LMDB/pool0 has replication-barrier timeouts also observed on current main. Core compiled/source regressions pass61 RocksDB /60 LMDB plus one existing skip; rebuilt baseline fails25 cases per engine.

Matrix details and qualifications

Node22.23.1/24.21.0/26.9.0 × pool0/2 × RocksDB/LMDB: eleven legs pass all8 tests. The remaining leg passed the strengthened limit case but timed out waiting for same-shard database rows; its targeted rerun timed out in setup. Bounded controls reproduce replication-barrier timeouts on main38897539/core9b8542 and the pinned pair, at different barriers; restoring only baseline Table.ts passes8/8. These probes call table.get, outside the changed collection filter. Retain all failed runs; no further retry or assertion change.

The initial local bd7587f/c1e192c7 matrix passed12/12 before merging Pro main. Compiled regression proof rebuilt both baseline and fix. Broad Pro CI build/unit/type/lint/integration jobs pass; companion-check awaits core#3571. Core's draft records its WebSocket mock, MQTT, design-budget and transient uWS CI qualifications. Both remain draft.

PR QA: skipped @ 1201ab2 — kept within this single-task run; author-run matrix exercises the real four-node route.

Docs: no companion; core restores local-only completeness and Pro adds regression coverage.

Refs #436.
Depends-on: harper#3571

🤖 Generated by OpenAI Codex; posted via @kriszyp.

Release target: v5.4 — main-line static-sharding regression; PR base is main.
Related PRs: #1116 overlaps, #1120 overlaps, #1086 overlaps, #1101 overlaps, #1092 independent, #1103 independent, #1112 independent, #1038 independent
Complexity: medium

Deferred findings

  • 6b32ed31e4301e29 nit test-coverage — integrationTests/cluster/staticSharding.test.mjs — Mixed-shard limit semantics remain uncharacterized — NOT FILED
Origin — the dispatch brief this PR was written from

Fix Harper-Pro main static-sharding limit regression after #1087

Dispatch: task harper-pro-main-red-static-sharding-20261022 · queued by unknown · ran by codex/gpt-6-astra/xhigh · worker kzyp-xps-1

Review-Coverage: authored=codex; ran=gemini,claude; adjudicated=domain; blocked=cursor-composer(failed); declined=cursor-grok,muse; rounds=6; full=3 @ 1201ab2

Review-Attention: read ~3m (decisions: pooled-query-contract, settlement-observation, do-less-alternative; raised: degraded review) @ 1201ab2

kriszyp and others added 4 commits October 26, 2026 06:54
Dispatch-Task: harper-pro-main-red-static-sharding-20261022

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: harper-pro-main-red-static-sharding-20261022

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Keep the strengthened static-sharding assertion and include the coordinated
runtime fix from HarperFast/harper#3571.

Dispatch-Task: harper-pro-main-red-static-sharding-20261022
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Include the computed-sort completeness repair in the coordinated core
draft HarperFast/harper#3571.

Dispatch-Task: harper-pro-main-red-static-sharding-20261022
Co-Authored-By: GPT-5 Codex <noreply@openai.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates static sharding integration tests and documentation to verify that limit queries load complete rows from pooled stubs without changing placement. It introduces a SettledReads resource and enhances LocalRows with additional metadata flags and residency information. Two issues were identified in the test fixture changes: first, new ArrayBuffer(4) is used where new SharedArrayBuffer(4) is required for Atomics operations to avoid a TypeError; second, existingEntry in _storeReplica can be nullish, which would cause a TypeError when accessing existingEntry.key in the finally block.

Preserve the current core ancestry and upstream replication repairs while
retaining the reviewed local-only fix and strengthened static-sharding test.

Dispatch-Task: harper-pro-main-red-static-sharding-20261022
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
@kriszyp
kriszyp marked this pull request as ready for review October 27, 2026 01:45
@kriszyp
kriszyp requested review from a team as code owners October 27, 2026 01:45
@kriszyp
kriszyp merged commit 4c8dc6f into main Oct 27, 2026
63 of 64 checks passed
@kriszyp
kriszyp deleted the fix/static-sharding-limit-regression branch October 27, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant