Repository navigation
[Store] Report why Client::Create failed - #4933
Open
xiaodouzi666 wants to merge 2 commits into
Open
xiaodouzi666 wants to merge 2 commits into
xiaodouzi666 wants to merge 2 commits into
Conversation
added 2 commits
October 23, 2026 19:16
The batch wrappers in RealClient (batch_get_buffer, get_into_ranges, batch_put_from, batch_upsert_from, batch_get_into and the three *_multi_buffers variants) run their metrics observer unconditionally, and the observer dereferences client_. When setup() failed or was never called, the *_internal functions return error results but the observer still runs and segfaults on the null client_. Return early from the observer when client_ is null. Add a test that calls each wrapper on a RealClient without setup(), and batch_get_buffer after a failed setup(). Refs kvcache-ai#4625.
Client::Create returned std::optional, so RealClient::setup() could only log "Failed to create client on port N" and return INTERNAL_ERROR (or INVALID_PARAMS when the port was given). The reason (master unreachable, version mismatch, transfer engine init failure) was only visible in earlier log lines from other components. Return tl::expected<std::shared_ptr<Client>, ErrorCode> instead. setup() now logs that error and returns it; with an auto-assigned port it logs the error of every attempt and returns the last one. The retry loop's comments no longer claim the failures are metadata registration conflicts. Existing callers compile unchanged except a test helper that named the optional type and a call to optional::reset(). Two test expectations follow the new codes: the failed-setup case in real_client_uninitialized_test checks for RPC_FAIL, and EgmConfigCannotFallbackWithoutMnnvl expects the transfer engine init error (INTERNAL_ERROR) instead of INVALID_PARAMS. Refs kvcache-ai#4625.
xiaodouzi666
requested review from
XucSh,
YiXR,
stmatengss and
ykwd
as code owners
October 25, 2026 21:07
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Refs #4625. Stacked on #4932 (skip transfer metrics for an uninitialized
client); only the second commit belongs to this PR.
Client::Createreturnedstd::optional<std::shared_ptr<Client>>, so afailure reached
RealClient::setup()with no reason attached:setup()logged onlyFailed to create client on port N, retry i/20, gave up after 20 tries andreturned
INTERNAL_ERROR.INVALID_PARAMS.The actual cause was printed earlier by another component, for example
master_client.cpp ... Client not availablefor an unreachable master,Version mismatchfor a client/master version skew, or the transfer engine'sown errors. In #4625 the reporter's log shows only the retry lines, so the
reason for 20/20 failures cannot be seen.
Changes:
Client::Createnow returnstl::expected<std::shared_ptr<Client>, ErrorCode>,the convention the rest of
ClientandMasterClientalready use. Its sixfailure exits return:
ConnectToMastererror;INVALID_PARAMSfor a malformed fsdir from the master (two exits);InitTransferEngineerror.setup()logs the error and returns it. With an auto-assigned port it logsthe error of every attempt and returns the last one. The retry count and
timing are unchanged.
registration conflicts". That is not what happens: the port in
local_hostnameis only a name suffix in the default P2PHANDSHAKE mode, andthe RPC metadata entry is overwritten, not checked. Those comments are
dropped or shortened.
Behaviour changes
setup()(C++ and Python): it used to beINTERNAL_ERROR(-1), orINVALID_PARAMS(-600) with an explicit port. Itis now the real code: for example
RPC_FAIL(-900) for an unreachablemaster,
INVALID_VERSION(-300) for a version mismatch, orINTERNAL_ERROR(-1) for a transfer engine install failure. Callers that only compare with 0
are not affected.
Failed to create client on port N: RPC_FAIL, retry 1/20andFailed to create client after 20 retries, last error: RPC_FAIL.Client::Createreturnstl::expected<..., ErrorCode>. Everyin-tree caller compiles unchanged (they use
operator bool/has_value()/value()/operator*), except two tests: a helper thatnamed the optional type, and a call to
optional::reset(). The return typeis not part of the mangled name, so out-of-tree C++ code built against the
old header must be rebuilt.
RealClientTest.EgmConfigCannotFallbackWithoutMnnvl(compiled only without
USE_MNNVL) now expects the transfer engine errorINTERNAL_ERRORinstead ofINVALID_PARAMS, which came from the old fixedreturn.
Suggested Merge Priority
Module
mooncake-transfer-engine)mooncake-store)mooncake-conductor)mooncake-reshard)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Built on Ubuntu 22.04 with g++ 11.4, Release, on top of
ac4d29bff2plus thefirst PR.
Test commands:
Test results:
Full build, including all store tests and e2e targets: 0 errors, and the
same 50 compiler warnings as
main.real_client_uninitialized_test: 20/20. Every run returnsRPC_FAILfromthe failed
setup()and logsFailed to create client after 1 retries, last error: RPC_FAIL.RealClientTest.EgmConfig*: 3/3.Full store ctest: 198/200, the same two pre-existing failures as
main(
TransferTaskTest.BatchGetOffloadObjectCopiesPinnedHostToGpu,MmapArenaTest.MixedAlignmentSequence); every test that passes onmainpasses here.
Python, against the built
_storeextension, with master127.0.0.1:1and
MC_STORE_CLIENT_SETUP_RETRIES=1:setup()returns-900(RPC_FAIL,logged as
last error: RPC_FAIL) instead of-1, andbatch_get_buffer(["k"])returns[None].Unit tests pass
Integration tests pass (if applicable)
Manual testing done (describe below)
Checklist
./scripts/code_format.shAI Assistance Disclosure
Claude Code was used to trace the code paths, draft the change, and run the
builds and tests. The submitter reviewed and understands every change.