feat: add moss bench command for latency benchmarking (closes #433)#448
Open
JITHENTHIRIYA wants to merge 1 commit into
Open
feat: add moss bench command for latency benchmarking (closes #433)#448JITHENTHIRIYA wants to merge 1 commit into
JITHENTHIRIYA wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
Please ensure that your PR meets the following requirements:
Description
Adds
moss bench, a new top-level CLI command that runs a query workload against an index and reports p50/p95/p99 latency percentiles. Users can supply queries inline(
--query, repeatable) or from a file (--queries-file, one query per line), control the number of timed runs (--runs, default 20) and warmup iterations (--warmup,default 3), and choose between local and cloud query modes. Output is a Rich table per-query plus an overall aggregate in human mode, or structured JSON with
--jsonforscripting and CI.
Fixes #433
Type of Change
Changes
packages/moss-cli/src/moss_cli/commands/bench.py— new command implementation with_percentile(linear interpolation, no extra dependencies) andbench_commandpackages/moss-cli/src/moss_cli/main.py— imports and registersbench_commandasmoss benchpackages/moss-cli/tests/test_bench.py— 15 tests covering_percentileedge cases, CLI validation errors, and happy-path scenarios (no credentials required)packages/moss-cli/README.md— adds### Benchmarksection under Commands and a feature bullet in the Features listExample Output
Loading index my-index locally...
Running 6 warmup + 40 timed iterations across 2 queries...
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Query ┃ p50 (ms) ┃ p95 (ms) ┃ p99 (ms) ┃ min (ms) ┃ max (ms) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ what is semantic search? │ 4.12 │ 5.88 │ 6.74 │ 3.91 │ 7.02 │
│ how does embedding work? │ 3.98 │ 5.21 │ 6.10 │ 3.75 │ 6.55 │
└──────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
Overall (40 runs)
p50: 4.05 ms
p95: 5.60 ms
p99: 6.48 ms
min: 3.75 ms max: 7.02 ms