Skip to content

Preserve full-text highlights when expiry changes the final visible page #3848

Description

@kylebernhardy

A code review of Allow stored-field sorting and exact counts for full-text search identified a timing-dependent gap between full-text highlighting and final Table materialization. Runtime reproduction is now confirmed: three genuine-TTL tests fail at 476386d1e8b4d392a66ef055fd26f756add52ea9 because a promoted visible row lacks highlights; all three pass with the final-page fix.

At reviewed revision 2c75b0d1c837c72d515bcd9992cf93e7a67fb259, the full-text index validates current source versions and expiry, traces only the requested page and releases its reader lease before Table finishes materialization. Table checks expiry again during result transformation. A selected row can expire between those stages; discarding it can promote a retained tail row outside the traced window, leaving that returned row without the requested $highlights.

The root fix needs coordination between Table's final eligibility/pagination and fulltext's reader-scoped tracing. Preserve Harper's ownership of record eligibility and the wrapper/Tantivy ownership of matching and highlighting. Tracing every retained tail row would defeat page-only highlighting and is not the intended workaround.

Acceptance:

  • Add a deterministic regression reproducing expiry between validation and final page materialization, including promotion of a retained tail row.
  • Trace the final returned page while preserving the same-reader guarantee and page-only work bounds.
  • Preserve cancellation, source-version validation and existing count/response contracts.
  • Cover sorted and exact-count paths that retain rows beyond the highlighted window.

Parent qualification work: Qualify native full-text filtering and catalog-scale operation for v5.4. The closed Reuse full-text highlight tracing without an extra continuation reader addressed reader reuse; it does not track this final-page eligibility race.

Priority rationale: P2, a contained timing-dependent incomplete-metadata result; no demonstrated data loss, outage or authorization bypass. Effort: Medium, because the known cause spans Table pagination and reader-scoped tracing and needs a deterministic timing harness. No release is committed. Introducing version/PR remains unverified; PR 3846 is the discovery context, not an asserted origin.

Review-Deferral: 28af9a8071dc7073 severity=nit category=correctness

Comment generated by kAIle (GPT-6)

Review-Deferral: c2a8334918923b07 severity=nit category=correctness

This is an additional review fingerprint for the same expiry-driven highlight promotion described above, not a second defect. The final-page fix in Allow stored-field sorting and exact counts for full-text search addresses both fingerprints; this issue remains open until that implementation merges.

Comment generated by kAIle (GPT-6)

Implemented fix and verification

Allow stored-field sorting and exact counts for full-text search now runs final eligibility through Table before tracing the final page under the original native reader lease. It reuses request-local selected records so expiry/projection are not repeated after tracing. It does not trace retained tail/offset rows. Offset eligibility remains necessary; visible projection and source refresh share the execution budget. A stalled refresh returns 503 and releases the lease, rather than returning an unhighlighted page.

Ten additional page-preparation regressions cover real native batch tracing, concurrent isolation, nested relationship restrictions, onlyIfCached expiry messages, error/cancellation identity and unresolved source refresh with actual reader retirement. Final focused source and compiled suites each pass 181 tests with four existing pending tests; deployed REST/docs: 27 pass. These tests do not establish catalog/load qualification.

Comment generated by kAIle (GPT-6)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions