Skip to content

[Store] Report why Client::Create failed - #4933

Open
xiaodouzi666 wants to merge 2 commits into
kvcache-ai:mainfrom
xiaodouzi666:fix/store-create-error-code
Open

xiaodouzi666 wants to merge 2 commits into
kvcache-ai:mainfrom
xiaodouzi666:fix/store-create-error-code

Conversation

@xiaodouzi666

Copy link
Copy Markdown
Contributor

Description

Refs #4625. Stacked on #4932 (skip transfer metrics for an uninitialized
client); only the second commit belongs to this PR.

Client::Create returned std::optional<std::shared_ptr<Client>>, so a
failure reached RealClient::setup() with no reason attached:

  • With an auto-assigned port, setup() logged only
    Failed to create client on port N, retry i/20, gave up after 20 tries and
    returned INTERNAL_ERROR.
  • With an explicit port, it always returned INVALID_PARAMS.

The actual cause was printed earlier by another component, for example
master_client.cpp ... Client not available for an unreachable master,
Version mismatch for a client/master version skew, or the transfer engine's
own errors. In #4625 the reporter's log shows only the retry lines, so the
reason for 20/20 failures cannot be seen.

Changes:

  • Client::Create now returns tl::expected<std::shared_ptr<Client>, ErrorCode>,
    the convention the rest of Client and MasterClient already use. Its six
    failure exits return:
    • the ConnectToMaster error;
    • the storage backend preparation error (two exits);
    • INVALID_PARAMS for a malformed fsdir from the master (two exits);
    • the InitTransferEngine error.
  • setup() logs the error and returns it. With an auto-assigned port it logs
    the error of every attempt and returns the last one. The retry count and
    timing are unchanged.
  • Two comments in the retry loop said the retries were for "metadata
    registration conflicts". That is not what happens: the port in
    local_hostname is only a name suffix in the default P2PHANDSHAKE mode, and
    the RPC metadata entry is overwritten, not checked. Those comments are
    dropped or shortened.

Behaviour changes

  • Return value of setup() (C++ and Python): it used to be
    INTERNAL_ERROR (-1), or INVALID_PARAMS (-600) with an explicit port. It
    is now the real code: for example RPC_FAIL (-900) for an unreachable
    master, INVALID_VERSION (-300) for a version mismatch, or INTERNAL_ERROR
    (-1) for a transfer engine install failure. Callers that only compare with 0
    are not affected.
  • Logs: Failed to create client on port N: RPC_FAIL, retry 1/20 and
    Failed to create client after 20 retries, last error: RPC_FAIL.
  • C++ API: Client::Create returns tl::expected<..., ErrorCode>. Every
    in-tree caller compiles unchanged (they use operator bool /
    has_value() / value() / operator*), except two tests: a helper that
    named the optional type, and a call to optional::reset(). The return type
    is not part of the mangled name, so out-of-tree C++ code built against the
    old header must be rebuilt.
  • Test expectation: RealClientTest.EgmConfigCannotFallbackWithoutMnnvl
    (compiled only without USE_MNNVL) now expects the transfer engine error
    INTERNAL_ERROR instead of INVALID_PARAMS, which came from the old fixed
    return.

Suggested Merge Priority

  • P0 — Urgent: Critical security vulnerability, data loss/corruption, or widespread production outage requiring immediate action
  • P1 — High: Significant correctness or stability issue in existing functionality, or a regression blocking users without a practical workaround
  • P2 — Normal: Routine bug fix, validated performance/compatibility improvement, or new feature for a confirmed use case
  • P3 — Low: Experimental feature, optional enhancement, refactoring, or cleanup with no urgent user impact

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake Conductor (mooncake-conductor)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Built on Ubuntu 22.04 with g++ 11.4, Release, on top of ac4d29bff2 plus the
first PR.

Test commands:

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DUSE_HTTP=ON \
  -DBUILD_UNIT_TESTS=ON -DUSE_CUDA=OFF -DWITH_STORE=ON -DWITH_STORE_RUST=OFF
cmake --build build
ctest --test-dir build/mooncake-store/tests --output-on-failure -j4
build/mooncake-store/tests/real_client_uninitialized_test --gtest_repeat=20
build/mooncake-store/tests/pybind_client_test --gtest_filter='RealClientTest.EgmConfig*'

Test results:

  • Full build, including all store tests and e2e targets: 0 errors, and the
    same 50 compiler warnings as main.

  • real_client_uninitialized_test: 20/20. Every run returns RPC_FAIL from
    the failed setup() and logs
    Failed to create client after 1 retries, last error: RPC_FAIL.

  • RealClientTest.EgmConfig*: 3/3.

  • Full store ctest: 198/200, the same two pre-existing failures as main
    (TransferTaskTest.BatchGetOffloadObjectCopiesPinnedHostToGpu,
    MmapArenaTest.MixedAlignmentSequence); every test that passes on main
    passes here.

  • Python, against the built _store extension, with master 127.0.0.1:1
    and MC_STORE_CLIENT_SETUP_RETRIES=1: setup() returns -900 (RPC_FAIL,
    logged as last error: RPC_FAIL) instead of -1, and
    batch_get_buffer(["k"]) returns [None].

  • Unit tests pass

  • Integration tests pass (if applicable)

  • Manual testing done (describe below)

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

Claude Code was used to trace the code paths, draft the change, and run the
builds and tests. The submitter reviewed and understands every change.

xiaodouzi666 added 2 commits October 23, 2026 19:16
The batch wrappers in RealClient (batch_get_buffer, get_into_ranges,
batch_put_from, batch_upsert_from, batch_get_into and the three
*_multi_buffers variants) run their metrics observer unconditionally,
and the observer dereferences client_. When setup() failed or was never
called, the *_internal functions return error results but the observer
still runs and segfaults on the null client_.

Return early from the observer when client_ is null. Add a test that
calls each wrapper on a RealClient without setup(), and batch_get_buffer
after a failed setup().

Refs kvcache-ai#4625.
Client::Create returned std::optional, so RealClient::setup() could only
log "Failed to create client on port N" and return INTERNAL_ERROR (or
INVALID_PARAMS when the port was given). The reason (master unreachable,
version mismatch, transfer engine init failure) was only visible in
earlier log lines from other components.

Return tl::expected<std::shared_ptr<Client>, ErrorCode> instead. setup()
now logs that error and returns it; with an auto-assigned port it logs
the error of every attempt and returns the last one. The retry loop's
comments no longer claim the failures are metadata registration
conflicts.

Existing callers compile unchanged except a test helper that named the
optional type and a call to optional::reset(). Two test expectations
follow the new codes: the failed-setup case in
real_client_uninitialized_test checks for RPC_FAIL, and
EgmConfigCannotFallbackWithoutMnnvl expects the transfer engine init
error (INTERNAL_ERROR) instead of INVALID_PARAMS.

Refs kvcache-ai#4625.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant