Add CI benchmark suite: latency + recall regression tracking

[Sravan1011]

Closes #435

What this adds

An automated benchmark harness (benchmarks/ci/) that runs on every push/PR to main, records p50/p95/p99 latency and recall@5/@10 per commit, and fails the build when results regress past configurable thresholds versus a checked-in baseline.

How it works

pytest benchmarks/ci/ -v \
  --benchmark-output=benchmark_results.json \
  --baseline-file=benchmarks/ci/baseline.json \
  --latency-threshold=0.20 \
  --recall-threshold=0.05

Stage	What it does
TestBenchmarkLatency	3 warmup rounds, then 20 measured rounds × 15 fixed queries (300 measurements) against a 1,000-doc subset of the shared benchmark corpus, moss-minilm, top_k=5
TestBenchmarkRecall	Compares returned doc IDs against pre-computed ground truth (ground_truth.json, generated with top_k=50); reports recall@5 and recall@10
TestRegressionGuard	Fails if P95 latency grows >20% or recall@5 drops >5pp vs baseline.json (both thresholds configurable via CLI flags)
TestWriteResults	Serializes commit SHA, timestamp, config, and all metrics to JSON

The GitHub Actions workflow (.github/workflows/benchmark.yml) uploads benchmark_results.json as an artifact (benchmark-results-<sha>, 90-day retention) on every run — including failed ones — so numbers accumulate per commit and can be published/graphed later.

Files

• benchmarks/ci/test_bench_ci_moss.py — the suite (latency, recall, guard, writer)
• benchmarks/ci/conftest.py — pytest CLI flags (--benchmark-output, --baseline-file, --latency-threshold, --recall-threshold)
• benchmarks/ci/generate_ground_truth.py — regenerates ground_truth.json when the corpus/model changes
• benchmarks/ci/ground_truth.json — committed ground truth for the 15 benchmark queries
• benchmarks/ci/baseline.json — performance baseline (placeholder for now, see below)
• .github/workflows/benchmark.yml — CI job with workflow_dispatch + update_baseline input
• benchmarks/ci/README.md — full usage docs; top-level benchmarks/README.md and AGENTS.md link to it

Verification

Tested end-to-end in a python:3.12-slim-trixie container (same Python as the workflow), invoked exactly as CI invokes it from the repo root:

• ✅ pip install -r benchmarks/ci/requirements.txt from repo root installs the local SDK + deps
• ✅ Full suite green: 4 passed, 1 skipped in 43s (skip = latency guard, expected with placeholder baseline)
• ✅ Measured: p50 ≈ 5.8ms, p95 ≈ 7.0ms (300 samples), recall@5 = recall@10 = 1.0 (15/15 queries)
• ✅ Failure path verified: with a deliberately bad baseline, both guards trip with clear diagnostics (P95 latency regressed by +181%, Recall@5 dropped by 0.0600 (threshold 0.0500))
• ✅ Results JSON is written even when guard tests fail, so the artifact upload (if: always()) has data
• ✅ No event-loop deprecation warnings (uses one explicit shared loop; safe on Python 3.14)

Notes for reviewers

1. baseline.json is intentionally a placeholder. Latency numbers are hardware-dependent, so the first real baseline should come from CI: trigger the workflow once after merge, download the benchmark-results-<sha> artifact, and commit it as benchmarks/ci/baseline.json. Until then the latency guard skips (recall guard is active but trivially passes against a zero baseline).
2. The update_baseline dispatch input does not auto-commit — the workflow has contents: read, so it copies the results over baseline.json on the runner and prints them; persisting is a documented manual commit. Happy to add an auto-commit step (needs contents: write) if maintainers prefer.
3. Fork PRs skip gracefully: without MOSS_PROJECT_ID/MOSS_PROJECT_KEY secrets the suite pytest.skips rather than erroring, so external contributors' PRs won't break — at the cost of not benchmarking them.
4. One transient moss-minilm model-download timeout was observed across many local runs; if it shows up in CI, pytest-rerunfailures would be a cheap hardening follow-up.

[cubic-dev-ai]

1 issue found across 12 files

_{You’re at about 97% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="benchmarks/ci/test_bench_ci_moss.py">

<violation number="1" location="benchmarks/ci/test_bench_ci_moss.py:196">
P2: Tests share a mutable `benchmark_results` dict and rely on implicit alphabetical class ordering for correct behavior (measurement → regression guard → writer). Consider using `@pytest.mark.order` or `pytest-dependency` to make the sequence explicit and resilient to random-ordering plugins or future renames.</violation>
</file>

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Tests share a mutable benchmark_results dict and rely on implicit alphabetical class ordering for correct behavior (measurement → regression guard → writer). Consider using @pytest.mark.order or pytest-dependency to make the sequence explicit and resilient to random-ordering plugins or future renames.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At benchmarks/ci/test_bench_ci_moss.py, line 196:

<comment>Tests share a mutable `benchmark_results` dict and rely on implicit alphabetical class ordering for correct behavior (measurement → regression guard → writer). Consider using `@pytest.mark.order` or `pytest-dependency` to make the sequence explicit and resilient to random-ordering plugins or future renames.</comment>

<file context>
@@ -0,0 +1,374 @@
+# ---------------------------------------------------------------------------
+
+
+class TestBenchmarkLatency:
+    """Measure end-to-end query latency over multiple rounds."""
+
</file context>

[Copilot]

Pull request overview

Adds a CI-oriented benchmark harness under benchmarks/ci/ to measure Moss query latency (p50/p95/p99) and retrieval stability (recall@5/@10) per commit, persist results to JSON, and optionally gate PRs/commits on regressions vs a checked-in baseline. This fits the repo’s broader goal of keeping the on-device runtime fast and stable across SDK/native changes.

Changes:

• Introduces a pytest-based benchmark suite that builds/loads a fixed CI index, measures latency + recall, compares to baseline.json, and writes benchmark_results.json.
• Adds a dedicated GitHub Actions workflow to run the suite on push/PR and upload per-commit benchmark artifacts (with a manual baseline-refresh mode).
• Documents the benchmark suite and links it from existing benchmark/agent guidance docs; updates ignores for generated outputs.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
benchmarks/README.md	Links the main benchmark docs to the new CI benchmark suite and describes behavior at a high level.
benchmarks/ci/test_bench_ci_moss.py	Core pytest suite: index setup, latency/recall measurement, regression guards, and JSON writer.
benchmarks/ci/requirements.txt	Minimal pinned deps for running the benchmark suite from repo root.
benchmarks/ci/README.md	Usage and operational documentation for running benchmarks, guards, and baseline updates.
benchmarks/ci/ground_truth.json	Committed ground-truth doc IDs per query for recall computation.
benchmarks/ci/generate_ground_truth.py	Script to (re)generate ground_truth.json when corpus/model changes.
benchmarks/ci/conftest.py	Adds pytest CLI flags for output paths and regression thresholds.
benchmarks/ci/bench_queries.py	Shared benchmark constants (queries, model id, index name, doc count).
benchmarks/ci/baseline.json	Initial placeholder baseline for regression comparison (recall active; latency intentionally zero).
benchmarks/.gitignore	Ignores CI benchmark __pycache__ and generated benchmark_results.json within benchmarks/.
AGENTS.md	Documents the new benchmark workflow in agent guidance.
.gitignore	Broadens .venv ignore pattern and ignores benchmark_results.json at repo root.
.github/workflows/benchmark.yml	New workflow that runs the benchmark suite and uploads results artifacts; supports manual baseline refresh behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.