Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,814 changes: 782 additions & 1,032 deletions crates/core/src/agent.rs

Large diffs are not rendered by default.

926 changes: 186 additions & 740 deletions crates/core/src/ledger.rs

Large diffs are not rendered by default.

18 changes: 8 additions & 10 deletions crates/core/src/prompt.rs
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ pub fn system_prompt_with_breakdown(project_root: &Path, registry: &Registry) ->
/// resent on every request for the life of the session.
const SELF_EXTENSION: &str = "\n\nWhen the user asks for a reusable capability (tool, skill, prompt template, hook, permission rule, provider, or memory), read its contract first with bash: openmax --spec <surface>. It gives the file path, format, approval, and activation rules; verify with openmax --check.\n\
Surfaces: tools|skills|prompts|hooks|permissions|providers|memory|stdio|mcp.\n\
Past sessions and memories: openmax --recall \"<query>\"; capability-file history and restore: openmax --ledger. Isolated or parallel work: a child openmax -p or openmax --stdio process, in tmux when it must outlive the turn.";
Past sessions and memories: openmax --recall \"<query>\". Isolated or parallel work: a child openmax -p or openmax --stdio process, in tmux when it must outlive the turn.";

/// One line per skill: name, description, and the SKILL.md path the model
/// reads on demand. Project skills show a project-relative path (read_file
Expand Down Expand Up @@ -414,10 +414,9 @@ mod tests {
assert!(prompt.contains("activation rules"));
assert!(prompt.contains("openmax --check"));
assert!(prompt.contains("openmax --recall"), "preserved history must be findable");
// The ledger is the only route to a capability file's approved
// versions and their restoration commands; neither --spec nor
// --recall exposes that, so the pointer has to.
assert!(prompt.contains("openmax --ledger"), "capability history must be findable");
// Capability-file history is no longer recorded, so a pointer to it
// would be bytes every request pays for a command with nothing to show.
assert!(!prompt.contains("openmax --ledger"), "no pointer to history that is not kept");
assert!(prompt.contains("openmax -p or openmax --stdio"));
assert!(prompt.contains("tmux"));
// The per-surface paths moved into the contracts the pointer names.
Expand Down Expand Up @@ -765,10 +764,9 @@ mod tests {
}

/// The path-free base rules, extension pointer, and builtin schemas must
/// fit in 3,500 bytes (the payload measured 3,412 bytes, 770 tokens on a
/// current tokenizer, when the cap was set, and 3,416 bytes, 773 tokens,
/// once the pointer named `mcp`). Grounding sections have
/// separate caps. Measure the payload with
/// fit in 3,450 bytes (the payload measured 3,357 bytes when the cap was
/// set, and 3,361 bytes once the pointer named `mcp`). Grounding sections
/// have separate caps. Measure the payload with
/// dump_frozen_prompt_payload_for_tokenizer and a real tokenizer before
/// changing this budget; provider framing is not included.
#[test]
Expand Down Expand Up @@ -796,7 +794,7 @@ mod tests {
.collect();
let tool_chars = serde_json::to_string(&builtins).expect("serialize").len();
let total = path_free + tool_chars;
const CAP: usize = 3_500;
const CAP: usize = 3_450;
assert!(
total <= CAP,
"frozen prompt budget exceeded by {} bytes: base rules + guide (path-free) \
Expand Down
2 changes: 1 addition & 1 deletion crates/core/src/recall.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
//! <path>"); the memory PR made facts durable. Recall is the searchable form
//! of the same commitment: what the harness preserved must be findable
//! without hand-grepping home-dir JSONL over bash. It is a read-only
//! standalone operation like `--ledger`: no session, no endpoint, no daemon,
//! standalone operation like `--check`: no session, no endpoint, no daemon,
//! and no derived index - the stores on disk stay the single source of truth
//! and every scan reads them directly. At harness scale (megabytes, not the
//! hundreds of millions of rows a database engine plans for) a bounded
Expand Down
72 changes: 47 additions & 25 deletions crates/core/src/registry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,17 +103,19 @@ pub struct Registry {
/// reason. Receipts and the unknown-tool error name these so a broken
/// write is never mistaken for a live capability.
pub broken: Vec<(PathBuf, String)>,
/// Every capability file path THIS freeze's capture actually read (tool
/// manifests and SKILL.mds). The refreeze classifier asks whether a path
/// still existed in this generation without a second disk probe, which
/// would race the capture it claims to describe. Empty for a
/// manifest-restored registry, which only ever sits on the outgoing side
/// of that comparison.
pub(crate) read_paths: std::collections::HashSet<PathBuf>,
/// Every capability file THIS freeze's capture actually read (tool
/// manifests and SKILL.mds), with a hash of the bytes it read. The
/// refreeze receipt compares two generations file by file from these
/// without a second disk probe, which would race the capture it claims
/// to describe. The session manifest carries them, so a resumed registry
/// compares the same way. None for built-ins only or a manifest written
/// before they were kept, which only ever sits on the outgoing side of
/// that comparison.
pub(crate) read_paths: Option<HashMap<PathBuf, u64>>,
/// Broken TOOL manifests with the name each occupies: the declared name,
/// or the file stem when the document is too broken to yield one - the
/// same derivation the withhold pass uses. Lets the refreeze classifier
/// see that a broken file already explains an absent name. Empty for a
/// same derivation the withhold pass uses. Lets the unknown-tool error
/// name the broken file that occupies a called name. Empty for a
/// manifest-restored registry.
pub(crate) broken_tools: Vec<(PathBuf, String)>,
/// Same-directory skill name collisions, one record per name (name,
Expand Down Expand Up @@ -166,16 +168,16 @@ pub(crate) struct ExtensionSnapshot {
tools_omitted: usize,
/// Skills discovered but dropped by the `MAX_SKILLS` index cap.
skills_omitted: usize,
/// Every capability file this generation read: (path, sha256, bytes).
/// The ledger records exactly this generation, so what it attests is what
/// the freeze actually used - never a second read that could differ.
pub(crate) files: Vec<(PathBuf, String, Vec<u8>)>,
/// Every capability file this generation read, with a hash of its bytes
/// taken where they were read: the receipt names a changed file from
/// this, so nothing downstream needs a copy of the bytes.
read_paths: Vec<(PathBuf, u64)>,
/// Files read but not loaded, with the reason. The bytes are already in
/// the fingerprint (a broken write still triggers a refreeze); keeping
/// the reason lets that refreeze's receipt say the tool is NOT live.
pub(crate) broken: Vec<(PathBuf, String)>,
/// The tool-tier subset of `broken` with the name each file occupies
/// (declared, or stem as the fallback), for the refreeze classifier.
/// (declared, or stem as the fallback), for the unknown-tool error.
pub(crate) broken_tools: Vec<(PathBuf, String)>,
/// Same-directory skill name collisions, one record per name: (name,
/// displaced paths, winning path, winner indexed). The receipt names
Expand All @@ -187,8 +189,7 @@ pub(crate) struct ExtensionSnapshot {
/// next refreeze or /reload. It is read at activation, not here: most
/// captures match the frozen fingerprint and are discarded, and a memory
/// scan in each would read every memory file and the whole access log
/// for nothing. Not ledger files (data, not capability), so not in
/// `files`.
/// for nothing. Data, not capability, so not in `read_paths`.
project_root: PathBuf,
}

Expand All @@ -206,7 +207,15 @@ pub(crate) fn capture_extensions(data_dir: &Path, project_root: &Path) -> Extens
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
let mut h = DefaultHasher::new();
let mut files_read: Vec<(PathBuf, String, Vec<u8>)> = Vec::new();
let mut read_paths: Vec<(PathBuf, u64)> = Vec::new();
// One file's content identity, for the refreeze receipt. The fingerprint
// still hashes the bytes itself, so the value persisted sessions recorded
// does not move.
let identity = |bytes: &[u8]| {
let mut file = DefaultHasher::new();
bytes.hash(&mut file);
file.finish()
};
// (path, reason, dir precedence index, declared name if recoverable):
// tools are keyed by DECLARED name, not file stem, so a collision between
// a broken file and a loaded definition must be judged on the name the
Expand Down Expand Up @@ -246,7 +255,7 @@ pub(crate) fn capture_extensions(data_dir: &Path, project_root: &Path) -> Extens
};
bytes.hash(&mut h);
let Some(bytes) = bytes else { continue };
files_read.push((path.clone(), crate::ledger::sha256_hex(&bytes), bytes.clone()));
read_paths.push((path.clone(), identity(&bytes)));
let Ok(text) = std::str::from_utf8(&bytes) else {
broken_at.push((path, "not valid UTF-8".into(), dir_index, None));
continue;
Expand All @@ -271,8 +280,8 @@ pub(crate) fn capture_extensions(data_dir: &Path, project_root: &Path) -> Extens
// under a valid project override: the override is legitimately active,
// and the reason says so instead of claiming the name is not callable.
let mut broken: Vec<(PathBuf, String)> = Vec::new();
// (path, occupied name) per broken tool file: the refreeze classifier
// tells "removed" from "explained by a broken file" with this, never by
// (path, occupied name) per broken tool file: the unknown-tool error
// names the broken file behind a called name with this, never by
// re-probing disk after the capture.
let mut broken_tools: Vec<(PathBuf, String)> = Vec::new();
for (path, mut reason, broken_dir, declared) in broken_at {
Expand Down Expand Up @@ -325,7 +334,7 @@ pub(crate) fn capture_extensions(data_dir: &Path, project_root: &Path) -> Extens
let bytes = std::fs::read(&path).ok();
bytes.hash(&mut h);
let Some(bytes) = bytes else { continue };
files_read.push((path.clone(), crate::ledger::sha256_hex(&bytes), bytes.clone()));
read_paths.push((path.clone(), identity(&bytes)));
let Ok(text) = std::str::from_utf8(&bytes) else {
broken.push((path, "not valid UTF-8".into()));
continue;
Expand Down Expand Up @@ -371,7 +380,7 @@ pub(crate) fn capture_extensions(data_dir: &Path, project_root: &Path) -> Extens
// read when the generation activates (`Registry::from_snapshot`). The
// directory path is still hashed so a project without memories keeps the
// fingerprint its persisted sessions recorded and resumes without a
// refreeze. Never ledgered (data, not capability).
// refreeze.
project_root.join(crate::memory::MEMORY_DIR).hash(&mut h);
let mut external: Vec<ToolSpec> = external_by_name.into_values().collect();
// Built-in shadows never load (assemble drops them); excluding them here
Expand Down Expand Up @@ -409,7 +418,7 @@ pub(crate) fn capture_extensions(data_dir: &Path, project_root: &Path) -> Extens
skills: discovered_skills,
tools_omitted,
skills_omitted,
files: files_read,
read_paths,
broken,
broken_tools,
shadowed_skills,
Expand Down Expand Up @@ -446,7 +455,7 @@ impl Registry {
registry.ext_fingerprint = snapshot.fingerprint;
registry.tools_omitted = snapshot.tools_omitted;
registry.skills_omitted = snapshot.skills_omitted;
registry.read_paths = snapshot.files.iter().map(|(p, _, _)| p.clone()).collect();
registry.read_paths = Some(snapshot.read_paths.into_iter().collect());
registry.broken = snapshot.broken;
registry.broken_tools = snapshot.broken_tools;
registry.shadowed_skills = snapshot.shadowed_skills;
Expand Down Expand Up @@ -503,7 +512,7 @@ impl Registry {
tools_omitted: 0,
ext_fingerprint: 0,
broken: Vec::new(),
read_paths: std::collections::HashSet::new(),
read_paths: None,
broken_tools: Vec::new(),
shadowed_skills: Vec::new(),
memory_files: None,
Expand Down Expand Up @@ -678,6 +687,12 @@ pub struct RegistryManifest {
/// sections.
#[serde(default)]
pub memory_rows: Option<Vec<(String, usize)>>,
/// Every capability file the freeze read, with the hash of the bytes it
/// read, so the first refreeze of a resumed session names each file that
/// changed while it was closed, a SKILL.md body edit included. Additive:
/// absent in older manifests, whose first receipt compares index lines.
#[serde(default)]
pub read_files: Option<Vec<(PathBuf, u64)>>,
}

/// Current manifest format. A manifest carrying any other version is treated
Expand Down Expand Up @@ -753,6 +768,12 @@ impl Registry {
// re-suspending must keep the accounting its persisted prompt
// still depends on.
memory_rows: self.frozen_memory_rows.clone(),
read_files: self.read_paths.as_ref().map(|read| {
let mut files: Vec<(PathBuf, u64)> =
read.iter().map(|(path, hash)| (path.clone(), *hash)).collect();
files.sort();
files
}),
version: MANIFEST_VERSION,
external_tools,
skills: self.skills.clone(),
Expand Down Expand Up @@ -796,6 +817,7 @@ impl Registry {
// the persisted prompt, which is precisely what the manifest carries.
registry.memory_files = manifest.memory_files.clone();
registry.frozen_memory_rows = manifest.memory_rows.clone();
registry.read_paths = manifest.read_files.map(|files| files.into_iter().collect());
registry
}

Expand Down
1 change: 1 addition & 0 deletions crates/core/src/sessions.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1884,6 +1884,7 @@ mod tests {
ext_fingerprint: 0,
memory_files: None,
memory_rows: None,
read_files: None,
});
assert!(load_manifest(&core, &id).is_none(), "and so does its manifest");
// All five session-scoped files, so this cannot regress one at a time.
Expand Down
4 changes: 2 additions & 2 deletions crates/core/src/spec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -929,8 +929,8 @@ provider failed with a rate limit, an overload, or a server fault; thinking
already streamed for that attempt is void), `diff` (call_id,
path, diff, added, removed), `approval_request` (approval_id, name, summary,
detail, reason, source_path, source_sha, and an optional `env`), `approval_settled` (approval_id,
outcome), `refrozen` (tools, skills, changes: the refreeze receipt naming
each recorded capability-file change and its actor), `schemas_over_budget`
outcome), `refrozen` (tools, skills, changes: the refreeze receipt, one line
per tool or skill file added, modified, or removed), `schemas_over_budget`
(schema_tokens, budget_tokens: the installed tools take most of what the
window can spend, so compaction runs early and stops entirely once they
reach it; advisory, at most once per session), `compacted` (tokens_before,
Expand Down
30 changes: 8 additions & 22 deletions crates/core/src/state.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,6 @@ use crate::config::Settings;
use crate::registry::Registry;
use crate::types::{AgentEvent, AgentEventEnvelope, ChatMessage};

/// One extension generation exactly as a freeze captured it: for each file,
/// its path, sha256, and bytes.
pub type ExtensionGeneration = Vec<(PathBuf, String, Vec<u8>)>;

/// In-memory state of one agent session.
#[derive(Default, Clone)]
pub struct SessionData {
Expand All @@ -58,17 +54,6 @@ pub struct SessionData {
/// context window. The condition holds on every turn once it holds at all,
/// so the advisory is emitted once and not per turn.
pub schemas_over_budget_reported: bool,
/// Whether the ledger has reconciled with the extension files this session
/// froze. False until the first turn start: a freeze reads disk directly,
/// so changes made while no session was running would otherwise never be
/// recorded - and the next mid-turn sync would sweep them up as the
/// agent's own work.
pub ledger_synced: bool,
/// A turn-start generation the ledger could not record. Held so a later
/// mid-turn sync lands it as external work first: the delta that sync
/// records would otherwise span a human's pre-session edits and file
/// them as the agent's. In memory only; see `agent::settle_ledger`.
pub unrecorded_external: Option<ExtensionGeneration>,
/// Content hashes of policy notices (inert allow rules, hooks that did
/// not load) already narrated to the MODEL this session. The condition
/// holds every turn once it holds at all, so the transcript gets one
Expand All @@ -80,9 +65,11 @@ pub struct SessionData {
/// seeded from the chain at build, then advanced as new events are
/// narrated. An approval recorded outside the running session (a human
/// at another terminal, #199) reaches it through no other channel - the
/// refreeze receipt names file changes, not approvals - so the turn
/// start names any it has not seen.
pub seen_ledger_events: HashSet<u64>,
/// refreeze receipt names file changes, not approvals - so a turn start
/// outside auto names any it has not seen. None until the chain was
/// first read: auto never consults it, so a session built in auto is
/// seeded by its first turn in another mode, which narrates nothing.
pub seen_ledger_events: Option<HashSet<u64>>,
/// Server-reported `prompt_tokens` over the local estimate of the same
/// request, as last observed on a turn's completion. The bytes/4
/// estimator under-counts BPE tokenizers on code, by far more than the
Expand Down Expand Up @@ -264,10 +251,9 @@ impl Core {
/// Settings say how to reach an endpoint and what a turn may spend. A
/// history search uses neither, so an unreadable settings file - a key
/// from a newer build, a hand edit, a stray comma - must not also make the
/// project's own history unreadable. `--ledger` already reads its store
/// without loading settings at all; this puts `--recall` on the same
/// footing, and leaves the fail-closed rule exactly where it earns its
/// keep: the paths that spend money and run tools.
/// project's own history unreadable. This leaves the fail-closed rule
/// exactly where it earns its keep: the paths that spend money and run
/// tools.
///
/// The failure is returned, never swallowed. Degrading silently to
/// defaults would hide a real misconfiguration behind a working search;
Expand Down
11 changes: 6 additions & 5 deletions crates/core/src/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -213,9 +213,10 @@ pub enum AgentEvent {
},
/// The session's tools, skills, and system prompt were re-frozen from
/// current config (extension files changed, or the user forced /reload).
/// `changes` is the refreeze receipt: one line per capability file the
/// ledger recorded, with who changed it ("tool.toml modified (external)"),
/// so the action space never mutates silently.
/// `changes` is the refreeze receipt: one line per tool or skill file
/// added, modified, or removed (".openmax/tools/deploy.toml modified"),
/// read from the outgoing and incoming registries, so the action space
/// never mutates silently.
Refrozen { tools: usize, skills: usize, changes: Vec<String> },
/// A forced compaction (`/compact`) finished. The automatic budget prune
/// speaks through the digest note it leaves in the transcript; the forced
Expand Down Expand Up @@ -398,9 +399,9 @@ mod tests {
env(AgentEvent::Refrozen {
tools: 7,
skills: 2,
changes: vec![".openmax/tools/deploy.toml added (session)".into()],
changes: vec![".openmax/tools/deploy.toml added".into()],
}),
r#"{"session_id":"s1","type":"refrozen","tools":7,"skills":2,"changes":[".openmax/tools/deploy.toml added (session)"]}"#
r#"{"session_id":"s1","type":"refrozen","tools":7,"skills":2,"changes":[".openmax/tools/deploy.toml added"]}"#
);
assert_eq!(
env(AgentEvent::SchemasOverBudget { schema_tokens: 6800, budget_tokens: 2150 }),
Expand Down
Loading
Loading