Skip to content

starts_with on an @indexed string silently misses rows with astral-plane (U+10000+) chars after the prefix — index range upper-bound 0xffff too low #1629

Closed

Description

@kriszyp

Summary

starts_with on an @indexed string attribute silently omits any row whose value continues past the prefix with an astral-plane (U+10000+) character (emoji, some CJK-extension, math/symbol blocks). Index-vs-scan divergence → silent wrong results.

Repro (harper 228eacc0f)

@indexed name: String. starts_with("zteam") via the index returns [ascii-control, bmp-control]; the true scan (oracle via contains, which is non-index-optimizable) returns [ascii-control, bmp-control, astral-probe] — the row name = "zteam" + 👨‍👩‍👧 (ZWJ family emoji) is missing from the indexed path only. (Equality, between, and PK identity are all byte-faithful and consistent — this is specific to starts_with.)

Root cause (code-confirmed)

resources/search.ts searchByIndex's starts_with case builds the index-range upper bound as:

end = value + String.fromCharCode(0xffff)

U+FFFF is only the max BMP char (UTF-8 = 3 bytes, leading byte 0xef). ordered-binary's writeKey merges UTF-16 surrogate pairs and emits supplementary-plane characters as 4-byte sequences with leading byte 0xf0+ — numerically greater than 0xef. So any indexed value continuing past the prefix with an astral char encodes to a key that sorts above the range's own upper bound, and the native range iterator never reaches it (the correct .startsWith() filter attached alongside can't help — the row is never scanned).

Fix

Use a byte-level maximum upper bound — append \u{10FFFF}, increment the key's last byte, or use ordered-binary's prefix-range upper-bound helper — instead of 0xffff.

Found via exploratory QA (verified against a same-semantics scan oracle; ruled out the canUseIndex reroute).

— KrAIs 🤖 (exploratory QA, on Kris's behalf)

Activity

  1. self-assigned this
    on Jul 21, 2026
  2. added 3 commits that reference this issue on Sep 22, 2026
    dee7529
    f561f8b
    ba4978c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

Priority

P1

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions