Skip to content

CI red on every PR: mcp 2.1.1 masks ToolError messages, breaking 5 test_server.py assertions #294

Description

@CryptoJones

Every test matrix job is failing on main-based PRs right now, on all four
OSes and every Python version. It is not any PR's fault — an unpinned upstream
dependency moved.

What fails

FAILED tests/test_server.py::test_missing_note_is_a_tool_error
  Expected regex: 'not found'
  Actual message: 'Error executing tool read-note'
FAILED tests/test_server.py::test_traversal_is_a_tool_error          ('path separators')
FAILED tests/test_server.py::test_edit_note_version_conflict         ('changed on disk')
FAILED tests/test_server.py::test_graph_neighbors_unknown_note_is_a_tool_error  ('not found')
FAILED tests/test_unified_graph_validates_operation_and_path_arguments          ('one of')

5 failed, 1013 passed. Coverage gate itself is fine (82.26% ≥ 80%).

Cause

pyproject.toml pins mcp>=2.0.0,<3.0. CI resolves that to the newest release
and installs mcp 2.1.1 (confirmed in the job log:
Downloading mcp-2.1.1-py3-none-any.whl). 2.1.x replaces the underlying
exception text with a generic Error executing tool <name> by the time it
surfaces as ToolError, so every pytest.raises(ToolError, match=...) that
asserts on the specific message now fails.

The tests still pass locally because the working venv has mcp 2.0.0 — which
is why this is invisible until CI runs. That version skew is the whole story.

Reproduction

Local venv (mcp 2.0.0):

$ .venv/bin/python -m pytest tests/test_server.py::test_missing_note_is_a_tool_error \
                             tests/test_server.py::test_edit_note_version_conflict -q --no-cov
2 passed in 0.64s

Same commit in CI (mcp 2.1.1): both fail as above.

Options

  1. Pin mcp>=2.0.0,<2.1 — restores green immediately, defers the real
    question. Worth doing first if something needs to merge today.
  2. Decide whether the message should still be reachable. These five tests
    assert that a caller can tell why a call failed — "not found" vs "path
    separators" vs "changed on disk" are meaningfully different, and the version
    conflict message in particular is what tells an agent to re-read before
    writing. If 2.1.x no longer surfaces that to clients, that is a real behaviour
    regression for the MCP surface, not just a test-fixture problem — worth
    checking whether the detail is exposed some other way (a structured error
    field, ToolError.__cause__) before relaxing the assertions.
  3. Relax the assertions to match the generic text — only if (2) concludes the
    detail genuinely is gone, and then it is worth noting what was lost.

I would not simply loosen the regexes without answering (2); these assertions are
protecting caller-visible behaviour, and the version-conflict message is load-bearing.

Adjacent

The dev environment should probably not drift this far from CI — the comment
above the pin in pyproject.toml shows the cap was set deliberately to catch a
breaking major, but a minor slipped through and only CI saw it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions