Skip to content

fix: preserve iterator snapshots and negotiate PK cursors - #616

Draft
xiaofan-luan wants to merge 1 commit into
milvus-io:mainfrom
xiaofan-luan:fix/iterator-first-page-pk-cursor
Draft

xiaofan-luan wants to merge 1 commit into
milvus-io:mainfrom
xiaofan-luan:fix/iterator-first-page-pk-cursor

Conversation

@xiaofan-luan

@xiaofan-luan xiaofan-luan commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

searchIterator capped pagination using a separate current-snapshot count. Remove that cap so a count from a different snapshot cannot stop iteration. Node already requested a real first page; keep using it for negotiation.

Add explicit search_iter_cursor_version=2 negotiation, typed raw-PK continuation and strict response validation. The default remains the existing distance cursor and ANN cost. An initial old-server response without the marker latches distance mode; a negotiated continuation cannot silently downgrade. No public protobuf version bump.

In negotiated PK mode, retain distinct identities across pages and pending cache so duplicate visible PK versions do not consume limit; advance the cursor using the final raw server row, including duplicate-only or filtered pages. Nonempty short pages are not EOF. Callback/validation failures do not skip the pending page. Identity memory is O(distinct PKs); external filters must retain row PK identity.

Node deduplicates retained raw protobuf IDs before the predicate and preserves its existing limit-before-filter semantics, including transformed display IDs and exact Int64/UInt64 validation.

Server companion: milvus-io/milvus#53965
Design: https://github.com/xiaofan-luan/milvus-design-docs/blob/089fbc90de99faa1d3eb3f21637d348fc2955d74/design_docs/20261005-search-iterator-pk-cursor.md
Related issue: milvus-io/milvus#50628 (comment)

Validation: 147 focused unit tests, tsc, and Prettier 3.6.2 pass. CI's prior system-test job failed pulling the MinIO image with unauthorized before SDK system tests started.

These are unit/mock checks. Complete-corpus enumeration requires negotiated server support and available raw vectors; BM25 live statistics and unsupported strict options are rejected by the server. Old-server fallback does not promise complete equal-score enumeration. Rolling-upgrade and broad storage/index end-to-end coverage remain follow-up validation. Keep this PR draft.

@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xiaofan-luan

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants