Do we need a Maka Experiments / Labs mechanism?

[me2seeks]

Maka is evolving quickly, and some changes are difficult to judge from implementation review or benchmarks alone. Users may need to try a behavior in real work, compare it with the stable behavior, and switch back before the project decides whether to keep it.

Today we do not have one user-facing way to offer that kind of trial. A change is usually either shipped as the product behavior, kept behind an ad hoc flag, or left for contributors to build locally.

Some examples of changes that could benefit from an opt-in trial are:

• collapsing the five Goal tools into one action-oriented Goal tool (Collapse five Goal tools into one action-oriented Goal tool #3622);
• trying different shell tool contracts, such as the Codex-style proposal in 将bash工具从Claude Code Style迁移到Codex Style #3342;
• presenting tool-schema information visually to vision-capable models;
• adding a persistent Node.js REPL tool.

These are examples only. This discussion is not intended to decide the design or adoption of any of them.

Question

Should Maka introduce an Experiments / Labs mechanism for users to enable selected experimental capabilities, try them in real workflows, provide feedback, and return to the stable behavior?

If we want this, a useful mechanism might provide:

• explicit opt-in, with experimental capabilities disabled by default;
• a clear description of what changes and what risks or limitations are known;
• a defined activation scope, such as the next Session rather than silently changing an active one;
• enough diagnostic identity to know which experiment and version produced a result;
• a straightforward way to disable the experiment and return to stable behavior;
• an owner and an exit decision: promote, revise, or remove it instead of keeping permanent feature flags.

This would be a product trial mechanism, not a replacement for @maka/eval. Eval answers reproducible benchmark questions; Labs would help collect real-world usability and compatibility feedback that benchmarks may not expose.

Questions to settle

1. Is a Labs mechanism useful for Maka at its current stage, or would it add more maintenance cost than evidence?
2. Should experiments be enabled from Desktop settings, CLI configuration, or both?
3. Should an experiment apply per Runtime Host, Workspace, Session, or another scope?
4. What minimum evidence should be required before an experiment becomes stable?
5. How should expiration and removal work so disabled experiments do not become permanent compatibility paths?
6. Should feedback be entirely user-submitted, or should Maka offer an explicit opt-in diagnostic bundle for an experiment?

The immediate decision here is only whether Maka needs this mechanism and what constraints it should have. Individual experimental features can continue to be discussed in their own threads.

AI assistance disclosure: Drafted and published with Codex at the author's direction.

[Astro-Han]

Thanks for framing this so carefully — especially the "not a replacement for @maka/eval" distinction and the insistence on an owner and an exit decision. Those are the right instincts, and the candidate list makes the need concrete.

My honest answer to Q1 is: the goal is real, but I don't think the mechanism clears its own bar yet. A Labs system is itself permanent machinery — a registry, a settings surface, diagnostic identity, exit tracking. The manager has to be cheaper than what it manages, and right now Maka has two facts that shrink what would be left for it:

1. The Desktop Nightly channel just landed (feat: add Desktop Nightly update channel #4108). "Let willing users try behavior in real work and report back" is exactly what a nightly channel provides, at zero new in-product machinery. A behavior flagged as experimental can ship to nightly, gather feedback, and either promote to stable or be reverted — with the weekly release cadence (four releases in the last ten days), the promote-or-revert loop is already short.
2. The strongest candidates are contract-level changes. Collapse five Goal tools into one action-oriented Goal tool #3622 and 将bash工具从Claude Code Style迁移到Codex Style #3342 change tool contracts that models are prompted against. Dual-running two shell contracts in production doubles the maintenance of one of the most security-sensitive surfaces in the repo, and the feedback that matters there ("does the model drive it well") is largely answerable by @maka/eval benchmarks rather than subjective trial.

On Q5, if Maka ever does adopt in-product trials, the one constraint I'd consider non-negotiable: expiration must be mechanical, not procedural. An experiment carries an expiry date in code and simply stops activating after it; "remember to remove the flag" as a process rule is how temporary branches become permanent compatibility paths. Per-session activation scope (Q3) would be the natural fit with that.

So: worth revisiting once nightly feedback shows a recurring class of changes that genuinely can't be evaluated there — at that point the evidence for the mechanism's own maintenance cost would exist too.

(Drafted with AI assistance; release cadence and nightly-channel claims verified against the repository.)

中文版本

感谢把这个想法整理得这么周全——尤其是"不替代 @maka/eval"的区分，以及对 owner 和退出决策的坚持。这些直觉都是对的，候选清单也让需求很具体。

我对 Q1 的诚实回答是：需求是真的，但这个机制目前还过不了它自己那关。 Labs 系统本身就是一套永久设施——注册表、设置界面、诊断标识、退出跟踪。管理者必须比被管理的东西便宜，而 Maka 当前有两个事实会压缩它能发挥的空间：

1. Desktop Nightly 通道刚刚落地（feat: add Desktop Nightly update channel #4108）。 "让愿意尝鲜的用户在真实工作中试用并反馈"正是 nightly 通道的用途，而且不需要任何新的产品内机制。实验行为可以先发 nightly、收集反馈，然后转正或回滚——按现在的周更节奏（最近十天四个版本），转正或回滚的回路本来就很短。
2. 最有分量的候选都是契约级改动。 Collapse five Goal tools into one action-oriented Goal tool #3622 和 将bash工具从Claude Code Style迁移到Codex Style #3342 改的是模型直接面对的 tool contract。在生产环境双跑两套 shell 契约，等于把仓库里安全最敏感的表面之一维护翻倍；而那里最重要的反馈（"模型能不能驾驭它"）很大程度上用 @maka/eval 基准就能回答，不必依赖主观试用。

关于 Q5，如果未来真的引入产品内试用，我认为有一条约束不可协商：过期必须靠机制，不能靠流程。 实验在代码里携带到期日，到期后自动停止激活；"记得删 flag"这种流程性约定，正是临时分支变成永久兼容路径的原因。按 Session 粒度激活（Q3）会是比较自然的搭配。

所以建议是：等 nightly 的反馈真的暴露出一类"在 nightly 里无法评估"的改动时再重启这个话题——到那时，这套机制自身维护成本的证据也就有了。

（本评论由 AI 辅助起草；发布节奏与 nightly 通道的论断均已对照仓库核实。）