[sssqqeer]

Description

Related: #4247.

transfer_sync_* and batch_transfer_sync_* can return while part of the batch is still running. As soon as one task fails, the sync wrapper frees the batch and either retries the whole batch or returns -1, while the other tasks keep reading the sources and writing the destinations.

In PD disaggregation this is silent KV corruption: prefill sees the failure, frees its KV blocks, and the next request's KV reaches the decode pages. The same happens when the outer MC_TRANSFER_TIMEOUT fires.

This PR waits for a failed batch to stop before acting on the failure:

• Before freeing a failed attempt, wait until every task is COMPLETED or FAILED, bounded by the existing deadline. Retry only after that.
• If tasks are still in flight at the deadline, return -2, the value of COMPLETION_UNKNOWN in the scatter ticket API: the buffers must stay alive. -1 now means "failed, and nothing touches the buffers".
• The success path and TENT are unchanged.

vLLM's MooncakeConnector checks ret == 0 and SGLang checks ret < 0, so both treat -2 as a failure.

A batch with a 4 KiB write that the target rejects and a 64 MiB valid write, over TCP. Right after the call returns, the caller writes new data into the source buffer, as it would when reusing it:

	main	this PR
rounds where the destination received data written to the source after the call returned	50/50	0/50

Suggested Merge Priority

• P0 — Urgent: Critical security vulnerability, data loss/corruption, or widespread production outage requiring immediate action
• P1 — High: Significant correctness or stability issue in existing functionality, or a regression blocking users without a practical workaround
• P2 — Normal: Routine bug fix, validated performance/compatibility improvement, or new feature for a confirmed use case
• P3 — Low: Experimental feature, optional enhancement, refactoring, or cleanup with no urgent user impact

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

cmake --build build
ctest --test-dir build -R python_sync_quiesce_test --output-on-failure
# existing initiator/target test, HTTP metadata server, TCP
python3 mooncake-wheel/tests/transfer_engine_target.py &
python3 mooncake-wheel/tests/transfer_engine_initiator_test.py

Test results:

• Unit tests pass: new python_sync_quiesce_test (mooncake-integration/tests/test_sync_transfer_quiesce.py); its failure-path test fails on main.
• Integration tests pass: transfer_engine_initiator_test.py 12/12 on both main and this branch.
• Manual testing done (describe below)

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (docs/source/api-reference/python/transfer-engine.md)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)