[ArnabChatterjee20k]

What

CPU embedding sessions ran with ONNX Runtime's default memory arena, which keeps the high-water mark of every input shape for the life of the session. A loaded nomic session therefore held its peak memory after every batch of long texts until idle unload kicked in.

CPU sessions now register ort::ep::CPU with the arena disabled. GPU providers are unchanged. ort is added as a direct dependency pinned to the version fastembed uses (=2.0.0-rc.13), so the lockfile still has a single ort.

This is the same fix that landed in claudear (appwrite/claudear#185), where it took a code indexing run from a 14.4 GB peak down to 3.8 GB. The pool already defaults to 1 here since #8, so only the arena part was missing.

Measured locally

Release builds of master and this branch, nomic, EMBEDDING_POOL_SIZE=1, 12 requests of 16 texts (100 to 1,800 words), idle unload off, macOS with jemalloc, 2 runs each:

	Peak RSS	Retained 3s after	Time
master	3.30 GB	3.30 GB	20.5s
this PR	2.01 GB	0.85 GB	20.9s

That is about 39% less peak and about 74% less retained memory, with no change in throughput. With 4 sessions and 4 concurrent requests, peak is unchanged (all sessions need their buffers at once) and retained memory was too noisy to call.

CI memory benchmark

Adds .github/workflows/memory-bench.yml and examples/bench_memory.rs (RSS via sysinfo, HTTP via ureq as a dev-dependency, so the server binary gains no dependencies). On every PR it builds the base and head binaries, runs the workload above against each (1 and 4 concurrent requests), and posts the table as a single PR comment that is updated on each push. Fork PRs only get the job summary since their token is read-only. The nomic model is cached between runs.

The benchmark only reports numbers; it does not fail the build.

🤖 Generated with Claude Code

[ArnabChatterjee20k]

@hansi-codes review

[github-actions]

Memory benchmark

Base 95b34e2 vs head fb7466c on ubuntu-latest.

Model nomic, EMBEDDING_POOL_SIZE=1, 12 requests of 16 texts (100 to 1,800 words), median of 2 runs. Retained is RSS 3s after the last request.

Concurrent requests	Peak (base)	Peak (head)	Δ	Retained (base)	Retained (head)	Δ	Time (base)	Time (head)
1	3089 MB	1772 MB	-43%	3048 MB	591 MB	-81%	151.3s	154.8s
4	3095 MB	1774 MB	-43%	3053 MB	594 MB	-81%	150.5s	152.6s

[ArnabChatterjee20k]

@hansi-codes review

[hansi-codes]

🟢 Tier S · Ready to merge

The updated test now checks the arena setting directly, and I found no outstanding issues in the new changes.

CPU embedding sessions now disable ONNX Runtime's memory arena by default, while explicitly configured execution providers remain unchanged. The PR adds a memory benchmark example and a GitHub Actions workflow that compares base and head RSS measurements and publishes the results. It also pins ort directly to the version used by fastembed and adds ureq as a development dependency for the benchmark.

Latest changes: The latest commit strengthens the CPU-provider test by downcasting to the concrete provider and checking that its debug output reports use_arena: false.

Verdict	New comments	Fixed	Still open
✅ Approved	0	1	0

📂 Walkthrough · 5

File	Change
Cargo.toml	Adds a pinned direct ort dependency and the benchmark's ureq development dependency.
Cargo.lock	Records the direct ort and ureq dependencies.
src/embedding.rs	Uses a CPU execution provider with the arena disabled when no providers are configured; tests that setting.
examples/bench_memory.rs	Adds an HTTP-driven workload that measures peak and retained process RSS.
.github/workflows/memory-bench.yml	Builds base and head binaries, runs the memory benchmark, and publishes its results in workflow output and eligible PR comments.

✅ Fixed since the last review · 1

• The test does not verify the arena setting · src/embedding.rs:897

---

_{Reviewed the commits since e27ddca · Details · Comment @hansi-codes review to re-run, or mention @hansi-codes with a question.}

[hansi-codes]

🔵 Tier A · Looks good to merge. Summary

[hansi-codes]

🟢 Tier S · Looks good to merge. Summary