Skip to content

feat: add moss bench command for latency benchmarking (closes #433)#448

Open
JITHENTHIRIYA wants to merge 1 commit into
usemoss:mainfrom
JITHENTHIRIYA:feat/moss-bench-command
Open

feat: add moss bench command for latency benchmarking (closes #433)#448
JITHENTHIRIYA wants to merge 1 commit into
usemoss:mainfrom
JITHENTHIRIYA:feat/moss-bench-command

Conversation

@JITHENTHIRIYA

@JITHENTHIRIYA JITHENTHIRIYA commented Jul 20, 2026

Copy link
Copy Markdown

Pull Request Checklist

Please ensure that your PR meets the following requirements:

  • I have read the CONTRIBUTING guide.
  • I have updated the documentation (if applicable).
  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

Description

Adds moss bench, a new top-level CLI command that runs a query workload against an index and reports p50/p95/p99 latency percentiles. Users can supply queries inline
(--query, repeatable) or from a file (--queries-file, one query per line), control the number of timed runs (--runs, default 20) and warmup iterations (--warmup,
default 3), and choose between local and cloud query modes. Output is a Rich table per-query plus an overall aggregate in human mode, or structured JSON with --json for
scripting and CI.

Fixes #433

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Changes

  • packages/moss-cli/src/moss_cli/commands/bench.py — new command implementation with _percentile (linear interpolation, no extra dependencies) and bench_command
  • packages/moss-cli/src/moss_cli/main.py — imports and registers bench_command as moss bench
  • packages/moss-cli/tests/test_bench.py — 15 tests covering _percentile edge cases, CLI validation errors, and happy-path scenarios (no credentials required)
  • packages/moss-cli/README.md — adds ### Benchmark section under Commands and a feature bullet in the Features list

Example Output

Loading index my-index locally...
Running 6 warmup + 40 timed iterations across 2 queries...

    Benchmark — my-index

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Query ┃ p50 (ms) ┃ p95 (ms) ┃ p99 (ms) ┃ min (ms) ┃ max (ms) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ what is semantic search? │ 4.12 │ 5.88 │ 6.74 │ 3.91 │ 7.02 │
│ how does embedding work? │ 3.98 │ 5.21 │ 6.10 │ 3.75 │ 6.55 │
└──────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

Overall (40 runs)
p50: 4.05 ms
p95: 5.60 ms
p99: 6.48 ms
min: 3.75 ms max: 7.02 ms

Review in cubic

@CLAassistant

CLAassistant commented Jul 20, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tooling: add a moss bench command to the CLI

3 participants