Repository navigation
docs(eval): publish post-merge baseline and next product fix - #72
jack-arturo wants to merge 8 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
jack-arturo
added
babysit:active
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f7749739ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Experiment Status | ||
|
|
||
| Generated: `2026-06-22T20:26:29+00:00` | ||
| Historical AMB snapshot: LoCoMo, PersonaMem, BEAM-100K values retained from the index generated 2026-06-22T20:26:29+00:00 because the raw archives are unavailable. These are historical values, not fresh AMB results. Source: origin/main@35fa73d, docs/experiments/index.json; full source commit/hash recorded per retained row in index.json. |
There was a problem hiding this comment.
The commit updates the generated STATUS/scoreboard to the October campaign and states that per-row provenance is available in index.json, but docs/experiments/index.json is unchanged from its June version: it contains no EXP-POSTMERGE-20261022 thread, artifact_overlay_root, or score_provenance fields. Any tooling or reviewer consuming the checked-in machine index therefore receives stale data that contradicts both generated views; commit the index produced by the same refresh run.
AGENTS.md reference: AGENTS.md:L95-L98
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The matching machine index and both generated human views are in stacked companion #73 at verified head e6cee41, with a current-head Codex 👍 and all review findings resolved. Its base is this branch at56b08e833b54c653bcbbafcb74f2379a5a51f05d, which also has a current-head clean review. Both descriptions require merging #73 INTO this branch FIRST, then merging #72 into main. Leaving this dependency finding open and #72 blocked until the prerequisite lands, so the registry, report and all three generated views reach main together. The baseline is complete. No merge or auto-merge has been performed.
jack-arturo
added
babysit:waiting-codex
|
Babysit stopped — |
jack-arturo
added
babysit:blocked
ChatGPT Codex Connector
|
Baseline batch COMPLETE. LongMemEval canaries: 21/30 each. Candidate full LongMemEval: 427/500 (85.4%; six-task macro 83.60%). Canonical LoCoMo: 1690/1986 (85.10%), 444 judge calls, zero errors/skips. All native and finalized accepted-window gates passed. Accounted $9.809397650 including $0.073130570 retained ambiguous liability; remaining $10.190602350 of the original $20. Twelve closed ledger copies and the sealed 222-file bundle verify. All owned paid runtime has been removed. Current #72 head 56b08e8 has a current-head Codex 👍. Fourteen local wrapper/history tests pass, including malformed-source rejection and the safe refresh hint. This repo has no benchmark CI workflow; its label-only helper is not counted as validation. BLOCKER class wait/dependency: the matching three generated dashboard files are in companion #73 (current head e6cee41), based on this branch. Merge #73 INTO test/postmerge-baseline FIRST, then merge #72 into main. The original P1 machine-index dependency remains open until that merge; do not merge #72 first. No GitHub merge or auto-merge has been performed. Selected product follow-up: automem-evals#31; automem#270 stays separate. |
jack-arturo
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c7a454e14
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
jack-arturo
added
babysit:active
jack-arturo
added
babysit:waiting-codex
jack-arturo
added
babysit:blocked
Publish the completed frozen comparison for control
01a17aaand candidate3eaa1f5. Both 30-question LongMemEval canaries score 21/30. Candidate LongMemEval-S scores 427/500 (85.4%; six-task macro 83.60%), and canonical LoCoMo scores 1690/1986 (85.10%), including 444 judge calls with zero errors/skips. Equal canaries and unchanged ranked retrieved IDs support no causal score-gain claim.Free paired diagnostics retain ForgetEval 234/385 and PMB 11/77 single, 0/12 session with no new assertion failures. Targeted lifecycle, tag and enrichment interleavings confirm write-path fixes. Full inline relations still expose obsolete content in all five supersede fixtures; automem-evals#31 is the selected next bounded fix, then one precision experiment and matched graph on/off. Keep automem#270 separately open and AMB/BEAM/adapters/architecture parked.
All four paid stages pass native exact-ID/contract/checkpoint gates. A disposable storage interruption quarantined 260 LoCoMo rows; the exact 497-row prefix was retained and the affected conversation rerun. Both finalized accepted windows (497+1489) audit clean. All failed and repeated costs remain included: $9.809397650 accounted, including $0.073130570 ambiguous liability, leaving $10.190602350 of the original $20. Twelve closed ledger copies match byte-for-byte. Official engineering metadata lives in verygoodplugins/automem#272; shared cost guard in #71.
The report, 494-row fixture audit and one completed registry record carry the product decision. Companion #73 refreshes all three generated dashboard files and preserves all 16 historical score records, including 12 populated results and their original configurations. Merge #73 into this branch FIRST, then merge this PR into main. This PR remains blocked until that prerequisite lands.
Validation: full native results and independent accepted-window audits passed; 34 offline metadata-builder and 24 projector checks passed; 14 dashboard-history tests passed, including exact history preservation, real-CLI malformed-source rejection and the documented safe refresh entrypoint. Earlier engineering suites remain recorded in the report. Production migration/repair is outside this batch. No merge or auto-merge enabled.
Implementation harness: Codex desktop. GitHub identity: configured connected account.