feat(ruvLLM): P1 persistent trajectory sidecar + P4 wiki-corpus pretraining#414
Open
Stricttype wants to merge 3 commits intoruvnet:mainfrom
Open
feat(ruvLLM): P1 persistent trajectory sidecar + P4 wiki-corpus pretraining#414Stricttype wants to merge 3 commits intoruvnet:mainfrom
Stricttype wants to merge 3 commits intoruvnet:mainfrom
Conversation
- examples/ruvLLM/Cargo.lock - examples/ruvLLM/Cargo.toml - examples/ruvLLM/docs/api-reference.md - examples/ruvLLM/docs/code-standards.md - examples/ruvLLM/docs/codebase-summary.md - examples/ruvLLM/docs/configuration-guide.md - examples/ruvLLM/docs/deployment-guide.md - examples/ruvLLM/docs/handoffs/2026-05-02-1943-auto.md - examples/ruvLLM/docs/project-overview-pdr.md - examples/ruvLLM/docs/system-architecture.md - examples/ruvLLM/docs/testing-guide.md - examples/ruvLLM/learn/260502-1900-init-ruvllm/learn-results.tsv - examples/ruvLLM/learn/260502-1900-init-ruvllm/summary.md Co-Authored-By: Pi Coding Agent <pi@localhost>
- examples/ruvLLM/Cargo.lock - examples/ruvLLM/config/example.toml - examples/ruvLLM/config/pretrain.toml - examples/ruvLLM/scripts/fetch-simple-wiki.sh - examples/ruvLLM/src/bin/pretrain.rs - examples/ruvLLM/src/bin/sidecar.rs - examples/ruvLLM/src/config.rs - examples/ruvLLM/src/lib.rs - examples/ruvLLM/src/sona/mod.rs - examples/ruvLLM/src/sona/persist.rs - examples/ruvLLM/src/training.rs - examples/ruvLLM/tests/persist_integration.rs Co-Authored-By: Pi Coding Agent <pi@localhost>
- rename src/data/ → src/corpus/ (gitignore conflict: data/ pattern blocks Rust source) - add corpus module: wiki corpus iter, tokenizer wrapper, tokenized dataset - add tests/wiki_pipeline_test.rs (5/5 PASS) - surgical fixes for pre-existing candle 0.8 API drift in src/inference_real.rs - add From<candle_core::Error> shim in src/error.rs (unblocks --features real-inference) - extend src/training.rs: DatasetSource trait, ModelCheckpoint serde, save_checkpoint, measure_baseline_perplexity P4 status: DONE_WITH_CONCERNS — pre-existing issues surfaced: - Trainer::train computes loss but does not call optimizer.step (no backprop) → perplexity-delta is structurally 0% until follow-up patch - SmallTransformer lacks from_checkpoint() constructor → trained checkpoints are saved but to_q4_weights() re-randomizes; follow-up needed to load saved checkpoint into inference path - TokenizerWrapper::from_pretrained stubbed (requires tokenizers/http feature, not currently enabled); inline whitespace WordLevel fallback works for offline pilot Smoke tests: - cargo check: PASS - cargo check --features persistence: PASS - cargo check --features real-inference: PASS - cargo check --features persistence,real-inference: PASS - cargo test --features persistence --test persist_integration: 4/4 PASS - cargo test --features real-inference --test wiki_pipeline_test: 5/5 PASS
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent foundation patches for ruvLLM (
examples/ruvLLM/), shipped together. Both are non-default — gated behind opt-in feature flags so existing builds (incl.no_std/ESP32) are unaffected.P1 — Persistent Trajectory Sidecar (feature:
persistence)Replaces the lossy in-memory
crossbeam::ArrayQueuetrajectory buffer with a durable SQLite-backed store. Fixes the SONA principle violation "no query is wasted" insrc/sona/trajectory.rs.src/sona/persist.rs—PersistentTrajectoryStorewith boundedmpscchannel + background writer threadNORMAL, schema versioning, drop-policy on overflowsrc/bin/sidecar.rs— minimal v1 stub (embedded thread; external-process upgrade is mechanical once IPC contract exists)P4 — Wiki-Corpus Pretraining Pipeline (feature:
real-inference)Curated cold-start replacing random-init perplexity baseline.
src/corpus/{mod,wiki,tokenize}.rs— Simple-English-Wiki dump reader, sharder, HF-tokenizers wrappersrc/training.rs—DatasetSourcetrait,ModelCheckpointserde,save_checkpoint/load_checkpoint,measure_baseline_perplexitysrc/bin/pretrain.rs— rewired with--corpusCLI flagscripts/fetch-simple-wiki.sh— pulls + extracts ~250MB dumpDisjoint by design
P1 touches
src/sona/*+ new sidecar binary. P4 touchessrc/corpus/*+src/training.rs+src/bin/pretrain.rs. Zero file overlap.Smoke tests (all green)
cargo check(default, persistence, real-inference, persistence+real-inference)cargo test --test persist_integration --features persistencecargo test --test wiki_pipeline_test --features real-inferencecargo build --bin ruvllm-sidecar --features persistencecargo build --bin ruvllm-pretrain --features real-inferenceKnown follow-ups (not in this PR)
Trainer::traincomputes loss but does not calloptimizer.stepagainst weight matrices — perplexity-improvement acceptance criterion is structurally blocked until this lands. Pipeline correctness validated; quality target deferred.Test plan
--features persistence--features real-inferenceno_stdbuild (persistence is gated)🤖 Generated with Claude Code