[sssqqeer]

Description

Part of #3030 (the lease-expired Get revalidation half).

A reader gets a time-based lease from GetReplicaList, transfers the bytes directly from the owner's segment, and then returns LEASE_EXPIRED (-707) if the lease ran out during the transfer. The check is purely time-based: it fails the read even when the object was never evicted, removed or rewritten. Transfers longer than the lease are common with TCP fallback, large batches and cross-rack reads, so whole batches come back as misses and get re-prefilled. This matches the -707 reports in #3076 and #4825. The only workaround today is a longer lease, which leaves fewer objects evictable.

This PR asks the master instead of failing outright. When a Get or BatchGet finishes after its lease expired, the client sends one batched BatchCheckReplicasLive(keys, {replica_id, write_version}). A key passes if:

1. the replica it read still exists. Replica IDs come from a monotonic counter and are never reused, so the buffer was not freed and handed to another object in between;
2. the replica is still readable (IsReplicaReadable, the same check GetReplicaList uses for new readers);
3. the object's write version has not changed. An in-place upsert rewrites the bytes under the same replica IDs, and the write version catches it.

Keys that pass keep their result. Keys that fail, or cannot be checked, get LEASE_EXPIRED as before. An older master does not return a write version, and the client then keeps today's behavior.

One host, TCP, master lease TTL set to 1 ms so every transfer outlives it. The same queried replicas are read once with today's behavior and once with this PR:

keys × size	today	this PR	this PR, half the keys removed and rewritten between query and read
64 × 4 MiB	64/64 LEASE_EXPIRED	64/64 OK	32 LEASE_EXPIRED, 32 OK
512 × 1 MiB	512/512 LEASE_EXPIRED	512/512 OK	256 LEASE_EXPIRED, 256 OK

Suggested Merge Priority

• P0 — Urgent: Critical security vulnerability, data loss/corruption, or widespread production outage requiring immediate action
• P1 — High: Significant correctness or stability issue in existing functionality, or a regression blocking users without a practical workaround
• P2 — Normal: Routine bug fix, validated performance/compatibility improvement, or new feature for a confirmed use case
• P3 — Low: Experimental feature, optional enhancement, refactoring, or cleanup with no urgent user impact

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

cmake --build build --target master_service_test client_integration_test
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata ./build/mooncake-store/tests/master_service_test
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata ./build/mooncake-store/tests/client_integration_test

Test results:

• Unit tests pass: master_service_test 82 passed, 2 skipped; client_integration_test 33 passed, 5 skipped (Ubuntu 22.04, Debug).
• Integration tests pass (if applicable)
• Manual testing done (describe below)

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)