Replies: 3 comments 5 replies
EnglishI agree with the central diagnosis and direction here. I implemented the original Runtime Host Artifact authority in #1503. That work established a single authenticated, lease-bound, crash-recoverable owner during the Host migration, but it did not establish that everything called an “Artifact” forms one product domain. The producer audit here is a good reason to revisit that boundary. The publication and purge-intent protocols protect one specific situation: metadata and filesystem bytes committed across two durability domains. They provide crash recovery, exact stable-ID replay, and safe cleanup after an uncertain commit. Operational state such as Session recap intent/result does not need that protocol. Session-owned blobs may still need equivalent guarantees, but they do not necessarily need the current generic Artifact authority or its shared source taxonomy. My preferred decomposition is:
I would not make “serve Tool Results directly from the RuntimeEvent log” part of this decision yet. #3630 identified valid reasons to keep the immutable execution ledger separate from the model-facing serving materialization: bounded reads, authorization, copied Sessions, remote Host placement, and independent retention. Retiring ArtifactStore does not require collapsing those responsibilities. Logical ownership and physical deduplication can be evaluated independently. A few details matter for sequencing:
That does not mean we need a long-lived compatibility mode. Our development workflow is AI-agent-driven, so re-homing these paths and completing a cutover can be substantially faster than a conventional migration. I would prefer a short, decisive migration over dual writes, parallel authorities, or an indefinitely retained legacy reader. Development speed, however, does not by itself make durable user state disposable. I suggest a one-time storage transition:
If the project deliberately decides that some Artifact classes are disposable, we can skip migration for those classes. That should be an explicit data-lifecycle decision, not an accidental consequence of removing enum values. I am comfortable with having no ongoing compatibility state; I am not comfortable with silently making an existing workspace prevent Host startup or lose user-visible deliverables. Removing the unused derivation producer can therefore be an independent, revertible first change. Removing legacy source values should happen as part of the one-time cutover. For this direction to close #4027 rather than merely relocate the cost, I think the cutover needs explicit performance invariants:
We can satisfy these requirements without a long-lived compatibility mode: migrate the load-bearing records during a versioned cutover, atomically retire the legacy authority, and reclaim obsolete payloads in bounded batches. The acceptance test should include a workspace with approximately 10–12k legacy Artifacts and verify that Host readiness and an unrelated Session retirement do not scale with the total Artifact count. So I support taking this direction to the dev list, framed as:
That preserves the useful invariants from #1503 without requiring unrelated data to continue sharing one lifecycle protocol. 中文我认同这里的核心诊断和总体方向。 我实现了 #1503 中最初的 Runtime Host Artifact authority。那项工作是在 Host 迁移期间建立一个经过认证、受 lease 约束、可崩溃恢复的统一 owner，但它并没有证明所有被称为 “Artifact” 的值属于同一个产品领域。这次 producer 审计足以说明我们应该重新审视这个边界。 publication 与 purge-intent 协议保护的是一个特定场景：metadata 与文件系统字节需要跨两个持久化域提交。它提供崩溃恢复、精确 stable-ID replay，以及不确定提交后的安全清理。Session recap intent/result 这类 operational state 不需要这套协议。Session-owned blob 可能仍需要同等保证，但不一定需要当前通用的 Artifact authority，也不应继续共享同一套 source 分类。 我倾向的拆分方式是：
我暂时不会把“直接从 RuntimeEvent 日志服务 Tool Result”纳入本次决策。#3630 已指出保持不可变执行账本与模型读取 materialization 分离的合理原因：有界读取、权限、Session 复制、远端 Host 部署以及独立 retention。退役 ArtifactStore 并不要求合并这些职责。逻辑 ownership 与底层物理去重可以分别评估。 推进时有几个细节很重要：
但这不意味着我们需要长期保留兼容模式。 我们的开发流程由 AI Agent 驱动，因此迁移这些路径并完成切换，速度可以明显快于传统开发。我更倾向于一次短促、明确的迁移，而不是引入 dual write、并行 authority 或无限期保留 legacy reader。 不过，开发速度本身并不会使用户磁盘上的持久化状态自动变得可丢弃。我建议执行一次性 storage transition：
如果项目明确决定某些 Artifact 类别本来就是可丢弃的，我们可以跳过这些类别的迁移。但这应当是显式的数据生命周期决策，而不是删除枚举值后意外产生的结果。我可以接受不保留持续运行的兼容态，但不能接受现有 workspace 因此无法启动 Host，或静默丢失用户可见的交付物。 因此，删除无人使用的派生 producer 可以作为第一个独立、可 revert 的改动；删除历史 source 值则应当与一次性 cutover 同时完成。 为了让这个方向真正关闭 #4027，而不是仅仅转移成本，我认为 cutover 还需要明确以下性能不变量：
这些要求不需要长期兼容模式：可以在 versioned cutover 中迁移仍然承重的记录，原子退役旧 authority，再以有界批次回收无用 payload。验收测试应包含约 10–12k 个历史 Artifact，并验证 Host Ready 与无关 Session 的退休耗时不会随 Artifact 总量增长。 因此，我支持把这个方向带到 dev 邮件列表，并建议将决策表述为：
这样既能保留 #1503 中仍然有价值的不变量，也不再迫使互不相关的数据继续共享同一套生命周期协议。 |
|
I think this discussion currently couples two decisions that should be separated. For #4027, the evidence supports a narrower diagnosis: That is primarily an
None of those steps requires first deciding the final product surface for Deep Research, uploads, or subagent write-backs. There is a separate and useful domain-boundary question here. Session recap state should not use a file publication protocol, and Write/Edit/Bash derivations without a reader should not be produced. But the presence of those misuses does not by itself prove that a shared physical blob engine must be retired. Logical ownership, typed facades, and physical storage are separate decisions. If the intended role of So I suggest splitting the scope:
The second effort is worthwhile, but it should not be a prerequisite for fixing the first, and #4027 alone does not justify a product-surface redesign. |
EnglishThanks — I think this is the missing distinction. I agree that we should separate:
The already-merged #4031 and #4033 are useful mitigations, but #4027 remains open and the measured residual still points to global metadata operations and readiness-path work. That evidence justifies indexed row/set operations, removing global scans from Host Ready, and bounding recovery work. It does not by itself justify redesigning Deep Research, uploads, or other product surfaces. I would push the clean target one step further, though. A bounded recovery journal is appropriate for repairing the current mutable, path-oriented store, but it may not be necessary in the zero-history design. With immutable blobs and typed references, the lifecycle can be:
A crash before the reference commit leaves a harmless orphan. A committed reference whose blob is missing is corruption and fails closed. Deletion no longer needs a cross-metadata/filesystem purge transaction, so publication and purge-intent journals may disappear rather than be carried into the replacement. I also think the resulting contract matters more than its name. A refactored Quota and retention should also be class-specific:
So I agree with splitting the work. #4027 should not wait for product-surface decisions. At the same time, I would avoid investing deeply in recovery machinery that the immutable target could eliminate. A useful first proof would be to put Read image snapshots and Tool Result archives behind the narrowed blob capability and verify:
Would this match what you mean by narrowing 中文感谢，这正是当前方案缺失的关键区分。我认同应该拆开三个决策：
已经合并的 #4031 和 #4033 是有价值的缓解，但 #4027 仍未关闭，测得的残留成本依然指向全局 metadata operation 与 readiness-path 工作。这些证据足以支持 indexed row/set operation、从 Host Ready 中移除全局扫描，以及限制 recovery 工作量；但它本身并不能推出必须重新设计 Deep Research、上传或其他产品界面。 不过，我想把干净终局再向前推一步。 若 blob 不可变，并通过 typed reference 使用，其生命周期可以简化为：
引用提交前崩溃只会留下无害 orphan；已经提交的引用找不到 blob，则属于数据损坏并 fail closed。删除不再需要跨 metadata/filesystem 的 purge transaction，因此 publication 与 purge-intent journal 可以被删除，而不是迁入新实现。 我也认为最终契约比名称更重要。无论叫重构后的 Quota 与 retention 也应该按 durability class 区分：
因此我同意拆分工作。#4027 不应等待产品界面决策；但同时也应避免在终局可以删除的 recovery 机制上投入过深。 一个合适的第一阶段证明，是让 Read 图片快照和 Tool Result archive 使用收窄后的 blob capability，并验证：
这是否符合你所说的将 |
|
Yes — this matches what I mean by narrowing the current authority, with one further simplification: before building another filesystem/S3-like immutable blob engine, I think we should test whether SQLite itself can be the physical substrate for the context-offload domain. Problem modelThe cohesive domain is not “Artifact”. It is Agent context offload:
Under that model, Read image snapshots and Tool Result archives belong together. Session recap effect state does not. Write/Edit/Bash copies without a reader should not exist. Uploads and durable user deliverables should not be forced into this store merely because they are bytes. The measured workspace is about 11.7k files / 6 GB, roughly 0.5 MB per payload on average. That is not a scale at which SQLite or a normal filesystem should require ten minutes to become ready. The failure comes from the access pattern: Proposed targetUse a dedicated CREATE TABLE context_blobs (
blob_id BLOB PRIMARY KEY, -- SHA-256
payload BLOB NOT NULL,
size_bytes INTEGER NOT NULL,
created_at INTEGER NOT NULL
);
CREATE TABLE context_refs (
ref_id TEXT PRIMARY KEY,
session_id TEXT NOT NULL,
owner_type TEXT NOT NULL, -- read_snapshot / tool_result_archive
owner_id TEXT NOT NULL, -- eventId or attachmentId
blob_id BLOB NOT NULL REFERENCES context_blobs(blob_id),
retention_class TEXT NOT NULL,
expires_at INTEGER,
created_at INTEGER NOT NULL,
UNIQUE(owner_type, owner_id)
);A put becomes one SQLite transaction:
SQLite WAL, The RuntimeEvent can commit its Reads remain bounded and authorized through This lets SQLite own the exact machinery we should not implement ourselves:
BoundaryThe material unknown is the payload size distribution, not the record count. We should measure p50/p95/p99/max and whether any context-offload producer requires streaming writes. Node If real payloads exceed that boundary, the credible fallback is hybrid storage: SQLite-inline context blobs below a threshold and an external immutable blob substrate only for proven large/streaming objects. I would not introduce that second durability domain before the distribution requires it. So my preferred framing is:
This is the same logical-ownership split you described, but it first asks whether we need to own a physical BlobStore implementation at all. |
EnglishThanks — after checking the SQLite and current Maka behavior more closely, I agree that SQLite should be the first physical substrate we test. However, one premise should be corrected before we commit to the proposed schema.
A local Node 26 / SQLite 3.53 probe confirmed the consequence: selecting 64 KiB from a 128 MiB BLOB raised peak RSS by approximately the full 128 MiB. With 512 KiB objects, SQLite-inline writes were faster than filesystem CAS plus a SQLite reference, and whole-object reads were comparable, but filesystem range reads were about 5× faster. This does not reject SQLite; it means the decision depends on the actual access contract. The current context consumers clarify that contract:
Therefore the current implementation is effectively a capped whole-object store, not a genuinely range-readable archive. If that is the intended contract, a monolithic BLOB in a dedicated SQLite database may indeed be the cleanest solution. We should state that reads are bounded by the per-blob limit, not by If we want Tool Result archives to become genuinely large and paged, I would test fixed-size chunk rows in the same SQLite database before adding an external filesystem substrate: context_blob_chunks(
blob_id,
chunk_index,
payload,
PRIMARY KEY(blob_id, chunk_index)
) WITHOUT ROWIDThat preserves one durability domain and atomic blob/reference commits while providing a real upper bound on per-query Node/SQLite materialization. An external immutable CAS remains appropriate only if measured workloads require substantially larger objects or streaming writes. Two other qualifications seem important:
So I would refine the proof in this order:
This still supports the #4027 target: indexed Session-scoped operations, no payload-tree scan, and no GC/vacuum/checkpoint on the readiness path. But #4027 identifies the broken access pattern; it does not by itself select the physical BLOB layout. My current preference is therefore:
中文感谢。进一步核查 SQLite 和 Maka 当前行为后，我认同应该首先验证 SQLite 作为物理底座。不过，在确定当前 schema 之前，有一个前提需要修正。
本地 Node 26 / SQLite 3.53 探针验证了这个影响：从 128 MiB BLOB 中读取 64 KiB，峰值 RSS 大约增加了完整的 128 MiB。对于 512 KiB 对象，SQLite inline 写入快于文件 CAS 加 SQLite 引用，完整读取性能相近，但文件 range read 大约快 5 倍。这并不否定 SQLite，而是说明选择取决于真实访问契约。 当前 context consumer 已经部分说明了这个契约：
因此，当前实现实际上是一个有大小上限的 whole-object store，而不是真正支持 range read 的 archive。如果这就是目标契约，那么 dedicated SQLite 中的单体 BLOB 很可能确实是最干净的方案；但应该说明读取由 per-blob limit 保证有界，而不是由 如果我们希望 Tool Result archive 真正支持大型对象和分页读取，我建议在引入外部文件底座前，先验证同一个 SQLite 数据库中的固定大小 chunk rows： context_blob_chunks(
blob_id,
chunk_index,
payload,
PRIMARY KEY(blob_id, chunk_index)
) WITHOUT ROWID这样仍然只有一个 durability domain，并保留 blob/reference 原子提交，同时真正限制每次查询在 Node 和 SQLite 中物化的数据量。只有实测证明存在更大的对象或 streaming write 需求时，才需要外部 immutable CAS。 另外还有两个需要明确的条件：
因此，我建议按以下顺序完成验证：
这仍然支持 #4027 的目标：indexed、Session-scoped operation；不扫描 payload tree；Host Ready 不执行 GC、vacuum 或 checkpoint。但 #4027 证明的是当前访问模式错误，本身不能决定物理 BLOB 布局。 因此，我当前更倾向于：
|
|
Agreed. The probe and consumer audit correct two assumptions in my proposal:
Given the current contracts, I think we can now make the narrower decision:
The existing 5 MiB Read-image limit and 4 MiB Tool Result archive limit provide the hard whole-object bound. The proof should therefore validate the retained producers by type:
The cross-database failure behavior is acceptable with those explicit bounds: committing the context blob/reference before the RuntimeEvent can leave an unreachable reference, but cannot create an Event that points to missing bytes. Best-effort cleanup, Session retirement, and enforced quota bound the leak; TTL must not independently delete replay-critical references. I would keep chunked SQLite as the next option only if we deliberately change the contract to support genuinely large paged archives. At that point chunks need independently verifiable integrity; otherwise whole-blob checksum verification would defeat bounded paging. External filesystem CAS should remain behind evidence that chunked SQLite cannot satisfy a real large-object or streaming workload. So the current decision ladder is:
This keeps the #4027 invariants unchanged: indexed Session-scoped operations, no payload-tree scan, and no GC, vacuum, checkpoint, or compaction on the readiness path. This direction looks decision-ready to me. |
EnglishI agree with the decision ladder, but I would stop just short of calling the monolithic SQLite design decision-ready. The desired contract can be “capped whole-object storage”; that is not yet the contract currently enforced by the producers, and the proposed schema still needs several properties required by #4027. 1. The existing 4 MiB archive limit is not a producer or storage bound
Therefore an archive above 4 MiB can currently be stored and replayed, but cannot be opened through
So the decision should say that we are introducing and enforcing an owner-specific whole-object bound, including oversize behavior and migration treatment for existing values. It should not treat the current 2.
|
|
Closing — this has moved into execution. The four separable needs are now tracked as #4027 (cold-start recovery sweep), #4037 (metadata rewrite), #4038 (batch purge), and #4071 (replace replay Artifact paths with a SQLite context). Thanks @likun666661. 简体中文关闭 —— 已进入执行阶段。四类需求分别由 #4027、#4037、#4038、#4071 跟踪。谢谢 @likun666661。 |
Uh oh!
There was an error while loading. Please reload this page.
Context
ArtifactStoreis a Runtime Host authority spanning 27 files / ~7.5k lines (excluding tests), centered on a 1888-line class that owns a SQLite metadata table, a filesystem layout, a writer lock, a bootstrap lock, a purge-intent marker protocol, publication staging with crash recovery, cross-record referential integrity, IPC, and a Desktop preview registry.#4027 reports the symptom: cold-starting a Host against 11.7k artifacts blocks readiness for ~10 minutes. Before optimizing that path, I'd like to ask what the authority is for — because reading the producers, "Artifact" turns out not to name one need.
Four unrelated needs currently share it
1. Model-replay bytes.
createReadImageSnapshotterstores image bytes theReadtool must re-feed to the model on replay;tool_result_archivestores oversized tool results so pruned turns can be re-read. Both are load-bearing. But the archive's content isserializeToolResultForArchiveover a RuntimeEvent payload that is already durably stored incore_agent_run_events— the same bytes, written twice, kept forever. #3630 asks the narrower version of this question.2. An effect journal that contains no file.
session-effect-coordinatorwritessession-recap-intent.jsonandsession-recap-result.jsonas artifacts, then recovers by reading the result artifact back whencreatefails. That is idempotent-transaction bookkeeping. It wants a row next to the other coordinator state, not staging files and a purge-intent protocol.3. Write amplification with no reader.
deriveToolArtifactCandidatescopies the entire file of everyWrite, a synthesized diff of everyEdit, and the redirect target of everyBash > fileinto the artifact store. Greppingsource: 'tool_result'across the repo finds write sites only — no consumer reads these back, andartifact-visibility.tshides them from the pane.provider_request_captureis in the same class. These are duplicates of bytes that already exist in the user's workspace, retained indefinitely, that nothing will ever open.4. User-facing deliverables.
deep_research,subagent_writeback,user_upload. The only ones a human sees.Two supporting facts:
ArtifactSourcevalues have no producer at all:synthesis_cache_block,history_compact_block,history_compact_source,export,snapshot,fixture— and three of those six are marked user-visible.ARCHITECTURE.mdmentions artifacts once, inside the eval result kernel. There is no stated product role for the Artifact store as a Host authority.Proposed direction
Retire
ArtifactStoreas an authority rather than optimize it.If this holds, most of the recovery protocol in
artifact-store.tsdisappears rather than getting faster, and #4027 becomes unreachable instead of mitigated.Questions to settle
tool_resultderivations that I failed to find?This retires a Host authority, so it is material. If the direction holds here, I will take the decision to
dev@maka.apache.orgbefore any implementation issue is opened; #4027 stays open in the meantime as the user-visible bug.AI assistance disclosure: I used Maka to audit the artifact implementation — producer/consumer greps, source-visibility mapping, and the recovery and purge paths. I reviewed every claim above against the source on current
mainand own its accuracy.中文
背景
ArtifactStore是 Runtime Host 的一个 authority，横跨 27 个文件 / 约 7.5k 行（不含测试），核心是一个 1888 行的类，同时持有：SQLite 元数据表、文件系统布局、writer lock、bootstrap lock、purge-intent 标记文件协议、带崩溃恢复的 publication staging、跨记录引用完整性校验、IPC，以及 Desktop 预览注册表。#4027 记录了症状：对着 11.7k 个 artifact 冷启动 Host，会阻塞就绪约 10 分钟。在优化这条路径之前，我想先问这个 authority 究竟为何存在——因为把生产者读一遍会发现，"Artifact" 并不指代同一个需求。
四个互不相干的需求共用了它
1. 模型重放字节。
createReadImageSnapshotter保存Read工具在重放时必须重新喂给模型的图像字节；tool_result_archive保存超大工具结果，好让被裁剪的轮次可以再读回来。两者都是承重的。但归档的内容是对 RuntimeEvent payload 做serializeToolResultForArchive的产物，而该 payload 已经durable 地存在core_agent_run_events里——同样的字节写两遍，且永久保留。#3630 问的是这个问题的窄版本。2. 一份不含文件的效果事务日志。
session-effect-coordinator把session-recap-intent.json和session-recap-result.json当作 artifact 写入，并在create失败时回读结果 artifact 来恢复。这是幂等事务记账，它要的是与其他 coordinator 状态并列的一行记录，而不是 staging 文件和 purge-intent 协议。3. 无人读取的写放大。
deriveToolArtifactCandidates会把每次Write的整个文件、每次Edit合成出的 diff、以及每条Bash > file的重定向目标复制进 artifact store。在全仓库 grepsource: 'tool_result'只能找到写入点——没有任何消费者读回它们，artifact-visibility.ts也明确把它们从面板隐藏。provider_request_capture属于同一类。它们是用户工作区里本就存在的字节的副本，被无限期保留，且永远不会被打开。4. 面向用户的交付物。
deep_research、subagent_writeback、user_upload。只有这一类是人真正会看到的。两条支撑事实：
ArtifactSource中有 6 个完全没有生产者：synthesis_cache_block、history_compact_block、history_compact_source、export、snapshot、fixture——而这 6 个里有 3 个被标记为用户可见。ARCHITECTURE.md只提过一次 artifact，在 eval result kernel 里。Artifact store 作为 Host authority 的产品职责从未被记录。主张的方向
退役
ArtifactStore这个 authority，而不是优化它。若此方向成立，
artifact-store.ts里大部分恢复协议是被删掉而不是被加速，#4027 也随之变为不可达，而非被缓解。需要定夺的问题
tool_result派生物是否存在我没找到的消费者？这会退役一个 Host authority，属于 material 变更。若方向在此处成立，我会在开出任何实现 issue 之前把决策带去
dev@maka.apache.org；在此期间 #4027 作为用户可见的 bug 保持开启。AI 使用披露：我借助 Maka 审计了 artifact 的实现——生产者/消费者 grep、source 可见性映射，以及恢复与清理路径。以上每一条论断我都已对照当前
main的源码核验，并对其准确性负责。All reactions