[kriszyp]

⊙ Problem

GET /Product/?score=gt=10&select(id,score) needs only the score index entries, but searchByIndex keeps only each entry's primary key, so every row is read and decoded anyway.

🤔 Is that the right requirement?

Yes, per Kris (2026-11-02/03): cover declared-scalar attributes and scalar/ID/untyped primary keys, include caching tables, and return expired rows until eviction. Accepted: unvalidated Date/BigInt integers decode per schema, ID/untyped primary keys read in index form, usedReadBytes drops (size-1 db-read), subscription initial scans hide expired rows, and covered caching selects return expired or invalidated-stub entries without refresh.

💡 Solution

Index entries now answer a select of the primary key, alone or with the scalar attribute its lone condition scans. 2,000 rows: 54–56% faster on RocksDB, 66–84% on LMDB.

The TTL scan now arms on catalog load and derives its interval from the table's TTL, not a one-day default. Partly addresses #3935 (eviction-only catalog scan arming and non-daily catalog scan intervals) and #2831 (leg 2 fixed; leg 1 visible in queries under threads: 0).

⚖️ Alternatives

Options considered

• Durable certificates and data-history markers: ruled out by Kris; the scalar gate reads only the declared schema.
• Per-row decision in transformEntryForSelect: rejected. The filter paths treat any {key, value} as a loaded record.
• multiGet: still decodes every record.
• pk-only coverage: misses select(id, attr).
• Queries never refresh expired cached rows: breaks stale-while-revalidate for queries and 11 caching tests.
• Splitting index keys before a surrogate pair: keys already on disk keep U+FFFD, so the reader must match it anyway.

🔧 Changes

• resources/Table.ts:
  • coveredSelectAttribute decides eligibility. Coverage stays off for replicateFrom: false (invalidated stubs keep index entries), @sharded tables (live baseline holds refuse rows one at a time), post-ordering, and row or vector guards. recordCoveredRead records the analytics.
  • coverableAttribute requires a declared SCALAR_TYPES type on secondary indexes and covers primary keys only when declared scalar, ID, or untyped; ID is excluded on secondary indexes because it admits an array of strings, and primary keys declared with an array type or elements (composite) take the record path (fixes 382d09518b5dba90 and feb518ca7dd020e2).
  • TTL scan root cause: every worker loads a runtime-created table from the catalog. The constructor arms TTL with the number form, which kept the one-day default interval. Only the creating thread, which does not own the scan, ran the object form. Both forms now derive the interval from scanInterval, or a quarter of expiration + eviction; a catalog-loaded eviction-only table arms the scan.
  • transformEntryForSelect no longer hides an expired record; a caching table still refreshes it unless only-if-cached or includeExpired.
• resources/search.ts:
  • executeConditions forwards coverage only to a lone AND lead.
  • admitCoveredEntries runs only with a filter or a declared-array dedupe. It settles overflow and -0 entries before distinctRecords, keeping main's per-entry yield. Look hardest here.
  • coverIndexEntries projects; declaredTypeDecoder restores declared Date and BigInt values.
  • overflowedValue matches the prefix in its stored UTF-8 form (toWellFormed), so a pair split by the 256-unit cut still matches. tsconfig.json adds ES2024.String.
• resources/DESIGN.md: expiry and covered-read rules; root index entry. resources/MAP.md: a pointer.

✅ Verification

New tests were red before their fixes and pass on both engines.

Tests, suites, A/B, follow-ups

• Runtime-created TTL table, 4 workers, RocksDB and LMDB: 10 rows reach record_count 0 with no query or GET (resolves 6e991ee3883f9f64, verified by PR QA qa-pr-harper-3938-8574f32 / P-6719: 100/100 rows evicted across 4 workers and after restart on RocksDB and LMDB). Red on 8a6f816 (10/10 stored after 30 s), green now. storage-reclamation.test.ts polls for eviction and passes; it shows TTL eviction, not pressure-driven reclamation.

• ttlCleanupInterval.test.js: a catalog-loaded table and the number form evict on schedule (red before the fix); an expired row stays queryable and stored until the scan evicts it; a configured scanInterval is kept. expiresAtAttribute's comment no longer says hydration skips eviction-only arming.

• The expiry, array-attribute and primary-key scalar-gate (composite and one-element array primary keys return arrays on select(id) and select(id, attr); declared ID and untyped primary keys stay covered; fixes 382d09518b5dba90 and feb518ca7dd020e2), and split-surrogate tests were red on the previous head; all pass on RocksDB and LMDB.

• Resources (RocksDB): 6414 passing, 0 failing. LMDB: 4884 + 795 passing.

• After the eviction fix (37e80cc): resources RocksDB 6417 passing; LMDB 4881 passing. Failures, all passing alone and unrelated to TTL: sourceApplyConflictRetry (RocksDB), 6 calibrationStore timeouts and 1 recordLockCoordinator (LMDB). vectorIndex stale-copy test now pins a long scanInterval: the faster scan could evict its row first and pass it vacuously.

• Other per-table maintenance: audit and txn-log retention are scheduled per store when the owner opens it (resources/auditStore.ts:474), so they were not affected.

• Follow-ups, not fixed here: create_table drops eviction (dataLayer/harperBridge/ResourceBridge.ts:165); scanInterval is not stored in the catalog, so a restart scans at the derived interval until the schema is redeclared (#2831); the RocksDB @expiresAt sweep (Fix RocksDB @expiresAt eviction sweeps); per-record expiresAt with no table TTL is best-effort (harper#1339); subscription initial scans still hide expired rows; an invalidated and expired row is a stub to a covered select but evicted by a wide one.

• Main unit, per directory: server 2344, security 1618, utility 1039, dataLayer 467, config 410, validation 346, sqlEngine 119, sqlTranslator 72. deployActivation and backupStreamFailure failed in one long single-process run and pass alone (75/75).

• Integration: resources + apiTests 1232 pass, 0 fail; database + server 1155 pass. Fails: 4 Ollama tests (need a local model).

• Integration not run: components, deploy, mqtt, mcp, upgrade.

• Updated visible-change pins: readOnlyModeTtl, nestedRelationshipConstraints (a covered caching-table select against the record path), onlyIfCachedExpiry, read-only-mode-ttl, apiTests/ttl.

• A/B (bench cases), RocksDB: covered id −56% (286 → 59 µs/q on indexRangeSelect.bench.js), id,score −54% (321 → 69 µs/q), indexed 291 → 59 µs/q, cold −78%. LMDB: −84%, −80%, −75%. 8 interleaved pairs, turbo off, bench-run. The LMDB select4-cold +15% in the full run is a case-ordering artifact: run alone it is −4% (CI includes 0).

• Single row (6 RocksDB / 4 LMDB pairs, 1 µs resolution): covered id,score 23 → 23 µs, 14 → 13 µs.

• Deferral (#4020): ordered-binary writes -0 as a corrupt key. Fix it at write in getIndexedValues and checkValidId. Repro: put(5, { value: -0 }), then value > 4 with select(id,value).

• Deferral (#4019): under snapshot: false, a long value rewritten mid-scan can leave select(id,value) but not select(id), and async overflow recovery after a yield reads through the snapshot (61571056b45cc528). Both go with the follow-up "overflow-key recheck reads latest state, not the query's snapshot" (primaryStore.getSync without the read transaction).

• Deferral: the scalar gate trusts validate; source fills and legacy integers written before a Date/BigInt declaration bypass validate (2977d59bbfab4c27, e89a67d47855fd83).

PR QA: Does a runtime-created TTL table keep evicting after a restart, with no query touching it? — passed @ 8574f32, qa-pr-harper-3938-8574f32

Docs: owner's separate task (querying, resource API and schema pages): scalar gate, expired query rows, one-element array ids.

Release note: TTL tables created at runtime or loaded from the catalog now scan every scanInterval, else (expiration + eviction) / 4, not daily.

Release target: main — a new query optimization with documented semantic changes, not a fix to backport
Related PRs: #3927 overlaps, #2155 independent, #3137 independent, #3307 independent, 3 others independent

🤖 Generated with Claude Code — Claude Opus 5.5

Deferred findings

• b74e1b3f336d1c25 minor performance — resources/search.ts — covered index-order sort yields a setImmediate and promise per row for a no-op filter — filed Async search iteration yields one event-loop turn per row #3241
• cb4e62a0c71f96f6 minor correctness — resources/search.ts — covered filter tests the raw -0 key shape before delegating to the record path — filed -0 is written to indexes and primary keys as a malformed ordered-binary key #4020
• c0867d378bb34ce5 minor correctness — resources/Table.ts — catalog-loaded tables scan at the derived quarter interval until a declaration applies scanInterval — filed Cleanup scans of loaded tables: none under threads: 0 until a write, and only daily on threads that load the table #2831
• 7086d236550d9817 minor correctness — resources/search.ts — uncovered overflow filter reads the latest committed value with getSync and no transaction — filed Long-string index filter reads the latest record instead of the query's snapshot #4019
• d0ba2ea426ba3a1a minor documentation — resources/DESIGN.md — no user-docs or release-note link for search returning expired rows while get hides them — NOT FILED
• 2c0b04e0a74564c0 nit documentation — resources/Table.ts — number-form tables now scan every (expiration+eviction)/4 instead of daily, with no operator note — NOT FILED
• 754573e389f1733e nit test-coverage — integrationTests/database/runtime-table-ttl-eviction.test.ts — runtime TTL test never asserts rows stay queryable between expiry and eviction — NOT FILED
• 648699a6d3a682f8 nit performance — resources/search.ts — HNSW pushdown lead does not receive covered — NOT FILED
• 2bc9766e48640550 nit performance — resources/Table.ts — Number/Any-typed primary keys never covered — NOT FILED
• 61571056b45cc528 nit correctness — resources/search.ts — unfiltered overflow under snapshot:false: select(id) skips recovery while select(id,value) drops a deleted or rewritten row — filed Long-string index filter reads the latest record instead of the query's snapshot #4019

Accepted behavior

Behavior the owner ruled intended, so there is nothing to file.

• 6b832a77a614bf34 major data-integrity — resources/Table.ts — covered select on a caching table returns the expired index value without refresh — accepted: Owner ruling 2026-11-03 option (a), in the PR body
• 6e991ee3883f9f64 major correctness — resources/Table.ts — expired rows on sourceless tables stay queryable where the cleanup scan never runs — accepted: Owner ruling 2026-11-02, in the PR body

6 minor or nit

• 1946b31e02242172 minor correctness — resources/Table.ts — initial subscription scan hides expired rows that search now returns — accepted: Owner ruling 2026-11-02, in the PR body
• 108201baa7107167 minor correctness — resources/Table.ts — ID/untyped primary key returned in index form by covered selects, stored form by loaded selects — accepted: Owner ruling 2026-11-02 and 2026-11-03, in the PR body
• 37c72520f069db9a minor correctness — resources/Table.ts — invalidated+expired row evicted by wide select, stub from covered select — accepted: Owner ruling 2026-11-02 and 2026-11-03, in the PR body
• 44ccc8a7be2a5987 nit correctness — resources/search.ts — untyped bigint id returned as number from covered rows and as bigint from -0-shaped record-path rows in one response — accepted: Owner ruling 2026-11-02, in the PR body
• 8584ce5735cb6052 nit reliability — resources/search.ts — overflow rows that read their record still record a size-1 db-read; SKIP recoveries record none — accepted: Owner ruling 2026-11-02, in the PR body
• 2977d59bbfab4c27 nit correctness — resources/search.ts — covered select decodes unvalidated legacy integers to Date/BigInt while the record path returns them as stored — accepted: Owner ruling 2026-11-02, in the PR body

Origin — the dispatch brief this PR was written from

searchByIndex's index branch discards the decoded index key (resources/search.ts ~1319, ({ value }) => value), so select: ['id'] or ['id', indexedAttr] over an index range still does N primaryStore.getEntry + record decodes (transformEntryForSelect / loadLocalRecord). Serve those selects from the index entry. Hazards to handle: (1) the candidate patch tracks hasNonScalarValues only at write time, so after a restart a table that already holds bigint/array/object/-0 values under an untyped indexed attribute would wrongly count as scalar: derive it at index open or restrict coverage to schema-typed scalar attributes; (2) check Date/Float/wrapper types round-trip through the index key; (3) keep residency and expiration gating (invalidated/expired rows must not be served from the index); (4) exercise LMDB as well as RocksDB. Report the A/B in the PR.

Acceptance

indexRangeSelect test (covered selects return identical rows to the uncovered path on rocksdb and lmdb, including after restart with non-scalar values present, expired and invalidated rows) red-without-fix where it asserts the read count; bench numbers in the PR body. Show each new test failing on origin/main sources (forced rebuild) before the fix.

_{Dispatch: task harper-covering-index-select-projection · queued by finding-triage/wave-43 · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=gemini; ran=cursor-composer,codex,cursor-grok,claude; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=29; full=5 @ 2446385}

_{Review-Attention: deep ~45m (critical: Table.ts, search.ts; decisions: search-returns-expired-rows, couple-expiry-change-with-covering, number-form-interval-derived, covered-gate-scope, bench-evidence, do-less-alternative) @ 2446385}

[gemini-code-assist]

Code Review

This pull request implements covered selects, allowing select queries to be answered directly from secondary index entries without loading full records from the primary store. It includes updates to documentation, benchmarks, and extensive unit tests. The review feedback highlights several critical issues in resources/search.ts and resources/Table.ts, including handling asynchronous promises during entry projection and deduplication, removing redundant setImmediate latency in the overflow resolver, preventing double-counting of database reads for recovered overflow values, and adding defensive optional chaining to prevent potential TypeErrors when accessing primaryStore.encoder.