Repository navigation
fix: preserve iterator snapshots and negotiate PK cursors - #616
Draft
xiaofan-luan wants to merge 1 commit into
Draft
xiaofan-luan wants to merge 1 commit into
xiaofan-luan wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xiaofan-luan The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
xiaofan-luan
force-pushed
the
fix/iterator-first-page-pk-cursor
branch
from
October 5, 2026 15:48
6d0456a to
c0798d5
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
searchIterator capped pagination using a separate current-snapshot count. Remove that cap so a count from a different snapshot cannot stop iteration. Node already requested a real first page; keep using it for negotiation.
Add explicit search_iter_cursor_version=2 negotiation, typed raw-PK continuation and strict response validation. The default remains the existing distance cursor and ANN cost. An initial old-server response without the marker latches distance mode; a negotiated continuation cannot silently downgrade. No public protobuf version bump.
In negotiated PK mode, retain distinct identities across pages and pending cache so duplicate visible PK versions do not consume limit; advance the cursor using the final raw server row, including duplicate-only or filtered pages. Nonempty short pages are not EOF. Callback/validation failures do not skip the pending page. Identity memory is O(distinct PKs); external filters must retain row PK identity.
Node deduplicates retained raw protobuf IDs before the predicate and preserves its existing limit-before-filter semantics, including transformed display IDs and exact Int64/UInt64 validation.
Server companion: milvus-io/milvus#53965
Design: https://github.com/xiaofan-luan/milvus-design-docs/blob/089fbc90de99faa1d3eb3f21637d348fc2955d74/design_docs/20261005-search-iterator-pk-cursor.md
Related issue: milvus-io/milvus#50628 (comment)
Validation: 147 focused unit tests, tsc, and Prettier 3.6.2 pass. CI's prior system-test job failed pulling the MinIO image with unauthorized before SDK system tests started.
These are unit/mock checks. Complete-corpus enumeration requires negotiated server support and available raw vectors; BM25 live statistics and unsupported strict options are rejected by the server. Old-server fallback does not promise complete equal-score enumeration. Rolling-upgrade and broad storage/index end-to-end coverage remain follow-up validation. Keep this PR draft.