Skip to content

fix(providers): surface OpenCode Zen free-tier refusal as actionable error - #716

Merged
rng1995 merged 1 commit into
NVIDIA:mainfrom
Yoseph-Zuskin:fix/opencode-zen-free-tier
Oct 5, 2026
Merged

rng1995 merged 1 commit into
NVIDIA:mainfrom
Yoseph-Zuskin:fix/opencode-zen-free-tier

Conversation

@Yoseph-Zuskin

@Yoseph-Zuskin Yoseph-Zuskin commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Fixes: #715

Problem

Zen answers every opencode_cli inference call with
FreeTierError 403 ("free tier can only be used from within OpenCode")
whenever the provider's deny-all isolation is present — so all semantic
scans via Zen free models fail 0/N with the opaque message
opencode exited with code 1; stderr='' and no hint of the cause.

Bisected to the single variable (each a live Zen call, sequential):

  • OPENCODE_CONFIG_CONTENT alone → 403; OPENCODE_PERMISSION alone → 403
  • Removing OPENCODE_CLIENT, file-based policy (opencode.json in the
    redirected config dir), paths/disables env groups → still 403
  • Vanilla config + default agent → OK; custom agent without permissions → OK
  • Model-independent: opencode/nemotron-3-ultra-free and
    opencode/mimo-v2.5-free both 403 under isolation, both OK without
    (mimo worked under full isolation on 2026-10-03, so this is a Zen-side
    policy change, not a provider regression)

Conclusion: ANY permission-deny trips the gate — the deny-all sandbox
and Zen free tier are mutually exclusive until Zen's policy changes.

Approach

Fail closed with guidance instead of the opaque exit-code message.
run_agent_cli now matches the FreeTierError envelope on a non-zero
exit and raises an actionable error (free-tier models incompatible
with the required sandbox; use a key-backed model or retry if Zen
policy changes). Deliberately NO silent fallback to a weaker sandbox.

Verification

  • 2 new cross-platform tests (fake Popen, no subprocess): refusal
    envelope → actionable error; plain non-zero exit → unchanged generic
    message
  • tests/provider/test_opencode_cli.py: 43 passed / 3 skipped
  • ruff check + ruff format --check clean, git diff --check clean

Risks

  • String matching on Zen's error envelope is brittle by nature; markers
    are a tuple (FreeTierError, can only be used from within OpenCode)
    so either signal fires, and a miss degrades to the previous generic
    message (never to a wrong success).
  • DCO: all commits carry Signed-off-by (maintainer: verify on push).

- run_agent_cli maps the Zen FreeTierError envelope (403 under the
  deny-all isolation, verified by env bisect, re-observed
  live 2026-10-03) to a fail-closed AgentCLIError naming the
  incompatibility and suggesting a key-backed model; plain non-zero
  exits keep the generic message
- No sandbox weakening: refusal never falls back to a weaker policy
- Verified: 2 new cross-platform tests (fake Popen); file suite
  43 passed / 3 skipped; ruff check + format + diff-check clean

Signed-off-by: Yoseph Zuskin <zuskinyoseph@gmail.com>
Co-Authored-By: OpenCode Muse Spark 1.3 Free (1M context) <noreply@opencode.ai>
@Yoseph-Zuskin Yoseph-Zuskin changed the title fix(providers): surface Zen free-tier refusal as actionable error fix(providers): surface OpenCode Zen free-tier refusal as actionable error Oct 4, 2026

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Hi @Yoseph-Zuskin, thank you for bisecting the Zen free-tier refusal and replacing a confusing exit-code error with useful guidance, without loosening the sandbox!

Value and readiness: This delivers what #715 asks for. Suppose opencode exits non-zero and the first 2,000 bytes of stdout contain FreeTierError or "can only be used from within OpenCode". run_agent_cli then raises an AgentCLIError that names the conflict with the sandbox and suggests a key-backed model. Every other non-zero exit keeps the generic message. Both paths still fail closed, and nothing falls back to a weaker policy. The check is limited to binary_name == "opencode". The tests cover both acceptance criteria and CI is green. Ready for final maintainer review.

Material findings

  1. [Non-blocking] src/skillspector/providers/_agent_cli.py:1253-1263: the markers are matched as plain substrings anywhere in the first 2,000 bytes of stdout. With --format json, stdout also carries the model's streamed text events, and the prompt contains untrusted skill content. A skill could make the model echo "FreeTierError". If the run then fails for an unrelated reason, the user is wrongly told to switch models. The call still fails closed, so only the diagnosis is affected. Matching only {"type":"error"} events, the envelope your test uses, would close this. An optional CliSpec hook for failure classification would also keep run_agent_cli CLI-agnostic, as the module docstring and the "HOW TO ADD A NEW AGENT CLI" note (:934) promise.
  2. [Non-blocking] src/skillspector/providers/_agent_cli.py:464-468: the comment states as permanent fact that Zen answers 403 to any call with the deny-all isolation. The record is mixed. #575 ran opencode/mimo-v2.5-free under this isolation on 2026-09-18. #613 hit the 403 on 2026-09-22. This PR reports mimo working under full isolation on 2026-10-03, before the refusal came back. Please date the observation, so readers treat it as Zen's current policy rather than a guarantee.
  3. [Non-blocking] The new message says "Use a key-backed model instead" but not how to choose one. Naming the setting, for example "set SKILLSPECTOR_MODEL to a provider/model you have credentials for", would make it fully actionable.

PIC tradeoffs: None identified. The PR deliberately refuses to relax the isolation for free-tier models, which matches the provider's fail-closed contract.

Verification and gaps:

  • Traced run_agent_cli at this head. The overflow and timeout checks still run first. The new branch only changes the message of an AgentCLIError that was already raised. stdout_snippet is used only for matching and never appears in the message.
  • Tests (tests/provider/test_opencode_cli.py:588-614): _FakePopen drives the real _run_bounded, and preflight=None is set through dataclasses.replace on the frozen spec. The refusal envelope produces the new message, and b"boom" keeps "exited with code 1". Two cases are not covered (optional): a marker beyond 2,000 bytes, and a non-opencode binary that prints the marker.
  • I could not confirm Zen's server behaviour. Per policy, I did not run the tests or OpenCode.
  • CI: all six checks pass. The PR merges cleanly with current main, #737, #738, #747 and #748.
  • Overlap: it conflicts with #713 in the constants block. #713 replaces _OPENCODE_SUPPORTED_VERSION, and this PR inserts _ZEN_FREE_TIER_MARKERS right after that line. The fix is mechanical: keep #713's bounds, then add the markers. I suggest landing this PR first, since #713 needs other changes and can rebase onto it.

Decision: Approved (reviewed head 56b900328f61568426acc130f15ef17e9f57dbb1)

@rng1995
rng1995 merged commit a5ba8b3 into NVIDIA:main Oct 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenCode Zen Free-Tier Refusal Under the Deny-All Sandbox (opencode_cli)

2 participants