feat(cli): bench init onboarding wizard + bench doctor by Yiminnn · Pull Request #883 · benchflow-ai/benchflow

Yiminnn · 2026-07-02T23:00:03Z

Adds first-run onboarding — the gap every user currently crosses by hand-exporting env vars and discovering broken creds mid-run.

Design researched against prior art (Harbor's flag-triple + credential-free oracle smoke; OpenClaw/Hermes/OpenCode's converged wizard-with-auth-branch + doctor split): Option B, wizard + doctor.

`bench init`

Interactive chain (every prompt has a flag mirror — fully-flagged runs never touch stdin):

Model → provider resolution (registered provider/model, bare family ids, or well-known families via infer_env_key_for_model).
Agent — offered list filtered by the same provider-protocol gate the run path enforces (_provider_supports_agent_protocol), so the wizard never offers an agent the run would reject (e.g. codex-acp on deepseek).
Task set / sandbox (dataset name or tasks dir; docker/daytona).
Credentials: host subscription login detected via check_subscription_auth → skip prompt; existing env var → reuse; else hidden prompt / --api-key → stored in ~/.benchflow/.env (0600, merge-preserving, auto-loaded by the CLI callback with setdefault semantics — a real export or CI secret always wins).
Smoke test: litellm route resolution row (pure, catches proxy-lane errors) + 1-token completion ping through the run path's own endpoint join (never GET /models — it can 200 while the route is broken; census evidence). ADC/AWS providers are skipped honestly. Optional --full-smoke --smoke-task X runs the credential-free oracle agent in the chosen sandbox first (Harbor's pattern).
Prints (and OSC52-copies) the canonical bench eval run … command.

`bench doctor`

Same checks standalone: per-row ✅/❌ (sandbox, provider key, litellm route, model ping), exit 1 on failure.

Quality

TDD-built: 13 red-green slices, then a 4-dimension multi-agent review with adversarial verification — 16 confirmed findings, all fixed (endpoint-join correctness for zai/github-models/azure-anthropic, corrupt-env-file crash of every subcommand, silent oracle-failure "Ready.", terminal-escape injection via server responses, --api-key silently dropped for ADC providers, and more — see the fix commit).
40 tests (tests/test_onboarding.py, tests/test_init_cli.py); full agents+cli suites green (135 passed).
Live-verified on a real box: init with real DeepSeek key → route row + ping 200; doctor green rc=0; interactive wizard over stdin; fake key → honest 401 + exit 1 with remediation.

TDD-built (red-green per slice) implementation of the researched Option B 'wizard + doctor split' design: - benchflow/onboarding.py — deep logic module: private env file (KEY=value, 0600, merge-preserving; setdefault autoload so a real export or CI secret always wins), TOML prefs round-trip, provider resolution (prefixed or bare model ids), agent picker filtered by the SAME provider-protocol gate the run path enforces (never offer an agent the run would reject), 1-token model ping (GET /models can 200 while the route is broken — a max_tokens=1 completion is the cheapest honest check), shared CheckResult rows for smoke + doctor, and eval-run command assembly (final command + credential-free oracle smoke argv). - cli/init_cmd.py — bench init: model -> agent -> tasks -> sandbox -> creds (reuse env / store hidden-prompted key) -> smoke -> prints + OSC52-copies the ready-to-run command; every prompt has a flag mirror so a fully flagged invocation never blocks on stdin. bench doctor: same checks as standalone pass/fail rows, exit 1 on failure. - cli/main.py — app callback auto-loads ~/.benchflow/.env (BENCHFLOW_HOME overridable) for every subcommand. 22 new tests (env file, prefs, provider resolution, protocol-filtered picker, mocked-transport ping, doctor composition, command assembly, non-interactive CLI, interactive wizard, startup autoload); full agents+cli suites green (117 passed).

Stage-1 of the researched two-stage smoke (Harbor's oracle pattern): with --full-smoke + --smoke-task, init runs the credential-free oracle agent on one task in the chosen sandbox BEFORE the credential checks — proving install + sandbox plumbing independently of keys. Opt-in because a real sandbox eval takes minutes; the default smoke (checks + 1-token ping) stays seconds.

Multi-agent review (correctness / silent-failures / security / design-fit, every finding runtime-repro'd by an adversarial verifier) on the bench init diff; all 16 confirmed findings fixed: - model_ping rewritten: endpoint joined exactly like the run path (resolve_base_url per protocol + one segment — no /v1 guessing, fixing zai //v4/v1 and github-models URLs), anthropic-messages-only providers ping /v1/messages with x-api-key (+ Azure api-key), ADC/AWS providers are skipped honestly instead of failed, 200-but-not-a-completion bodies fail, and all server-supplied detail text is control-character-sanitized (terminal escape injection). - env file hardening: 'export ' prefixes and single-quoted values parse; malformed lines (empty/whitespace keys) are skipped; unreadable files warn-and-degrade — a corrupt ~/.benchflow/.env can no longer crash every subcommand from the autoload callback (including the ones needed to fix it). - run_doctor: unknown sandbox is a failing row (was: zero checks, "All checks passed"); new pure litellm-route resolution row catches proxy-lane errors the direct ping cannot; unregistered-but-inferable models (claude-*/gpt-*/gemini-*) check their inferred key instead of hard-fail. - init guards: stage-1 oracle failure now fails init (was silently "Ready."); --full-smoke without --smoke-task errors (exit 2, was silent skip); --api-key on non-api_key providers errors with the real auth mechanism (was silently discarded); the smoke verifies the key just provided, not a stale exported one; host subscription logins (check_subscription_auth) are detected and skip the key prompt; bare well-known models complete the wizard via infer_env_key_for_model. 17 new tests (40 total for init/onboarding); full agents+cli suites green (135 passed). Live re-verified: real deepseek ping 200 on the corrected join + litellm-route row.

… bugs) Second review round (comment-analyzer, pr-test-analyzer, silent-failure- hunter, type-design-analyzer, code-reviewer) on top of the earlier adversarial pass. Confirmed behavioral bugs fixed: - The wizard's default dataset emitted a command that DOESN'T RUN: bare 'skillsbench' fails bench eval run's <name>@<version> parsing (and --full-smoke then blamed the sandbox). Default is now skillsbench@1.1 and registry-style names are validated via parse_dataset_spec at init time. - A non-UTF-8 byte in ~/.benchflow/.env bricked EVERY subcommand (UnicodeDecodeError is a ValueError, not OSError — the autoload callback died before any command, including the repair paths). read_env_file now degrades with a warning naming the file. - write_env_file destroyed comments/unparseable lines on rewrite (data loss introduced by the crash fix). The merge is now line-preserving (replace keys in place, keep everything else verbatim) and refuses to clobber an undecodable file. - Subscription-auth setups were failed by their own smoke test: run_doctor now takes the agent and, when check_subscription_auth covers the model's key, reports a skipped row instead of red key/route/ping rows. - bench doctor tracebacked on corrupt or partial config.toml (TOMLDecodeError / KeyError); now the same "run `bench init`" remediation, exit 1. - --full-smoke tracebacked with FileNotFoundError when `bench` wasn't on PATH; now a clear message. - CheckResult gains skipped=True (rendered "○", counted in the doctor summary) so all-green cannot over-certify unverifiable setups; the anthropic ping sends the required anthropic-version header; the litellm route row names the exception type; the modal sandbox row no longer says "unknown" on success; stale-exported-key shadowing warns; env-write OSError during init exits cleanly instead of dumping a traceback over a just-typed key. Comment corrections per the accuracy review (credential precedence, "pure functions", resolve_provider custom-endpoint rot, run-path-join wording, SigV4 -> Bedrock bearer, help texts) and test hygiene (the full-smoke test no longer makes a live network call; RUF005; obfuscated assertion; resolve_provider return annotation; skipped-flag coverage). 52 init/onboarding tests green; agents+cli suites green (95 passed).

UX redesign per review — choose, don't type (the Hermes setup pattern): - Every step is a numbered menu with an Enter-able default: provider (from the registry, with model-family hints), model (from the provider's model catalog when it has one, else a hinted prompt), agent (the protocol- filtered list), task set (live entries from the dataset registry with descriptions, newest first, plus a local-tasks-dir option and a free-form fallback when the registry is unreachable), sandbox (from SANDBOX_PROVIDERS). Free text survives only as explicit "other" options and CI flags (all flag mirrors unchanged). - After the model is chosen the wizard finds credentials ITSELF (onboarding.detect_key): host subscription login > process environment (which includes the saved ~/.benchflow/.env via the startup autoload) > ./.env in the working folder — a key found there is announced and passed through into the bench setup for future runs. The hidden prompt is now the last resort, not the default. New onboarding seams, TDD-built: detect_key (4 tests: precedence + pass-through), dataset_choices (registry fetch, newest-first, offline degrade), plus a full CLI test driving the menu flow with zero key typing. 59 init/onboarding tests green; agents suite green (70 passed). Live: Enter x4 + model id -> key auto-found in ./.env -> real 1-token ping 200 -> runnable command.

Yiminnn · 2026-07-03T06:00:00Z

UX redesign (commit `1fbfb59`): choose, don't type

Per review feedback — the wizard is now selection-driven (the Hermes setup pattern):

Provider:                       Task set:
  ...                             1) skillsbench@1.1 — 87-task benchmark set ...
  5) deepseek — deepseek          2) skillsbench@1.0 — superseded by 1.1 ...
  ...                             3) a local tasks dir — path to your own tasks
  16) other — type a model id   Select [1]: ⏎
Select [5]: ⏎
Model id (e.g. deepseek-...): deepseek-v4-flash
Agent (able to route ...):    Select [33]: ⏎   # protocol-filtered list, default pi-acp
Sandbox: 1) docker 2) daytona 3) modal          Select [1]: ⏎

✓ DEEPSEEK_API_KEY found in ./.env — saving it to your bench setup.

Every step is a numbered menu with an Enter default — provider (registry + family hints), model (provider catalog when present), agent (protocol-filtered), task set (live dataset-registry entries w/ descriptions, newest first; offline → free-form fallback), sandbox. Free text only via explicit other options; all CI flag mirrors unchanged.
Credentials are auto-detected after model selection (onboarding.detect_key): subscription login → process env (incl. the saved ~/.benchflow/.env) → ./.env in the working folder (announced + passed through into the bench setup). The hidden prompt is the last resort, not the default.

59 tests green (incl. a full menu-driven CLI test with zero key typing) · live-verified: Enter×4 + model id → key auto-found in ./.env → real 1-token ping 200 → runnable command.

The agent (the thing the user came to benchmark) is now the first menu; the provider menu then narrows to what that agent can route via the new onboarding.compatible_providers — the same protocol gate as compatible_agents, applied in the other direction (codex-acp offers only Responses-capable providers; deepseek disappears). compatible_agents(model= None) returns the unfiltered list for the first menu; the post-model consistency gate still guards flag combinations. Tests updated + a new menu-filter test (60 green); live-verified: Enter x2 + model id -> key from ./.env -> ping 200.

- "a local tasks dir" answered with a bare relative name (mytasks) fell into the registry-spec validation and hard-exited, discarding every wizard answer. Normalized at the prompt site to ./name so it routes to --tasks-dir. - The ./.env pass-through persisted and immediately spent a possibly unrelated key with only a post-hoc note. Interactive runs now get an Enter-default confirm; the message names the absolute source, the destination file, and a last-4 fingerprint (non-tty stays unprompted so CI never blocks). - detect_key ordered subscription > exported key — the OPPOSITE of the run path (resolve_agent_env inherits the exported key and uses_native_subscription_auth then defers to it), so init could announce "subscription login, no key needed" while the run billed the key. Precedence now matches the run path: environment > subscription > ./.env. - dataset_choices crashed on non-dict registry entries instead of degrading; malformed entries are now filtered. 63 init/onboarding tests green; full suites 133 passed.

…clack UI Fixes from live user testing of the wizard: - The agent menu showed only the 9 core built-ins. agent_paths() now triggers the remote-manifest auto-load and groups the FULL catalog (core + benchflow-ai/agents manifests + installed plugins) by the naming law; the wizard asks path first (acp / ai-sdk / omnigent, with counts), then that path's agents — the user's preferred shape for a ~60-agent list. - Provider labels lied: they showed each provider's PRIMARY wire next to an agent that would use a different endpoint (aws-bedrock read "openai-responses" beside claude-agent-acp; the filter was right — bedrock and zai genuinely serve anthropic-messages — but the label wasn't). Labels now show model families, "BYO base URL" for vllm-style providers, or the endpoint the CHOSEN agent will actually use. - Subscription login is now a LISTED choice, not a silent auto-decision: the credentials step is an OpenClaw-style menu of every detected source (subscription login, environment key, ./.env key — fingerprinted, with absolute paths) plus manual entry; interactive-only (_isatty seam), the non-tty path keeps run-path-preference auto-detection so CI never blocks. detect_key_sources() returns all sources in run-path order; detect_key is now its first-hit wrapper. - Clack/OpenClaw-style visuals via the existing rich console (no new dependency): ◆ step headers, │ option rails, ● confirmed-step summary lines, dim descriptions; degrades to plain text on non-tty; user data escaped against rich markup. 68 init/onboarding tests green (paths classification, matched-protocol labels, auth menu w/ subscription choice, source ordering); full suites 138 passed. Live pseudo-tty run verified end-to-end incl. the credentials menu.

…talog Adversarial review round on the wizard v3 (4 confirmed findings): - --agent naming a catalog-only agent exited with a bogus "wire protocol mismatch": the flags path never triggered the manifest autoload the menus use. It now resolves through registry.resolve_agent (aliases + the same miss-driven autoload as `bench eval run`) and unknown agents say "Unknown agent", not protocol mismatch. - The credentials menu announced choices it didn't honor: picking the subscription login with a key exported left the key in force (the smoke verified the DECLINED key, and the run would bill it); picking ./.env under a different exported key was a setdefault no-op. Choices are now effective in-process (subscription pops the key so the smoke verifies the subscription setup; ./.env/manual picks override) with an explicit shadow warning that the shell export wins at run time — shared _warn_shadow helper with the --api-key branch. - Offline with a warm cache, the wizard silently shrank to built-ins even though the cached clone holds the full catalog: _source_root now falls back to the cached benchflow-ai/agents checkout when the refresh fetch fails (the clone is the catalog of record). Skipped: a fetch timeout in benchmark_repos (shared with task sources — separate change if wanted). - A dim "loading the agent catalog…" notice before the first menu (the autoload can hit the network). 72 init/onboarding tests green; agents suite green.

…nment pick ty flagged use[1] (str | None) flowing into str-typed sinks — the ./.env branch now binds a narrowed value; the environment pick also gets its ✓ confirmation line (it was silently branch-less).

bingran-you

Daily scan review (2026-07-03): requesting changes. This PR belongs to benchflow-ai/benchflow and CI is green, but the onboarding flow has user-facing correctness blockers that need fixes before merge.

Findings:

Credential detection does not match the actual run path. _wizard_auth_step() consumes detect_key_sources() where process env wins over ./.env, but resolve_agent_env() reads ./.env before os.environ. The wizard can validate/smoke one key while the printed bench eval run later uses another.
Fully flagged local task-dir onboarding is not a true mirror of the prompt path. --dataset mytasks is rejected unless the user writes ./mytasks, while the interactive prompt normalizes the same bare directory answer.
Azure Foundry OpenAI smoke pings use Authorization: Bearer for all openai-completions providers, but Azure OpenAI API-key auth requires api-key, so bench init / bench doctor can falsely fail valid Azure setups.
Printed commands use raw string joining, so paths with spaces are not copy-pasteable.

Please align auth-source precedence with the runtime resolver, normalize flagged dataset paths the same way as the prompt path, use the provider-specific Azure auth header in smoke checks, and shell-quote printed commands. Keep the current tests, but add focused regressions for those cases.

bingran-you · 2026-07-03T12:29:49Z

Users Simulation review (2026-07-03): still blocked.

The basic init/doctor smoke is healthy: a temp-home interactive bench init --skip-smoke with a preseeded DEEPSEEK_API_KEY, patched local agent catalog, and TTY path wrote config/env successfully; bench doctor passed with the final ping stubbed. Targeted checks also passed: tests/test_init_cli.py tests/test_onboarding.py tests/test_agent_env_resolution.py tests/test_resolve_env_helpers.py -> 175 passed, plus ruff, ty, and help for bench, bench init, bench doctor, and bench eval.

But the existing changes-requested blockers on this same head are still visible in source:

Azure OpenAI smoke still uses Authorization: Bearer for openai-completions pings instead of Azure's api-key header (src/benchflow/onboarding.py).
Printed commands still use raw string joining, so paths with spaces are not copy-pasteable (src/benchflow/onboarding.py, src/benchflow/cli/init_cmd.py).
The fully flagged path still rejects bare local task dirs such as --dataset mytasks before normalizing them like the prompt path (src/benchflow/cli/init_cmd.py).

No model-backed trajectory was generated in this smoke, so trajectory health was not applicable.

bingran-you · 2026-07-07T13:22:14Z

Re: #883 (comment) and #883 (comment)

Pushed c4a73d637798d5ba7abda0216ff2d14d3bbd0d00 to feat/bench-init.

Changes:

model_ping() now sends max_completion_tokens: 1 for GPT-5 / o-series OpenAI-compatible chat-completions pings, while keeping max_tokens: 1 for older chat-completions models. This addresses the live Azure Foundry OpenAI GPT-5 rejection (max_tokens unsupported).
Added/updated the PR feat(cli): bench init onboarding wizard + bench doctor #883 regression coverage so the Azure GPT-5 ping asserts api-key auth plus max_completion_tokens and no max_tokens.
Split the oversized tests/test_init_cli.py into focused modules: non-interactive/doctor/smoke remains 393 lines, interactive/credential-source behavior is 474 lines, catalog/package-agent behavior is 195 lines, and shared CLI helpers are 29 lines. The split preserved the 43 existing init CLI tests with no missing/extra test names.

Local verification:

uv sync --extra dev --locked -> success
uv run python -m pytest tests/test_onboarding.py tests/test_init_cli.py tests/test_init_cli_interactive.py tests/test_init_cli_catalog_agents.py -q -> 100 passed
uv run python -m pytest tests/test_task_download.py tests/test_daytona_dind_compose_up.py -q -> 36 passed
uv run ruff check src/benchflow/onboarding.py tests/test_onboarding.py tests/test_init_cli.py tests/test_init_cli_interactive.py tests/test_init_cli_catalog_agents.py tests/init_cli_helpers.py -> passed
uv run ruff format --check src/benchflow/onboarding.py tests/test_onboarding.py tests/test_init_cli.py tests/test_init_cli_interactive.py tests/test_init_cli_catalog_agents.py tests/init_cli_helpers.py -> passed
uv run ty check src/benchflow -> passed
git diff --cached --check -> passed before commit

GitHub current-head verification is green on c4a73d637: test, pip-audit, parity, detect-scope, and rollout-smoke all passed; rollout-smoke completed in 9m45s.

I am moving labels from status:blocked / review:changes-requested to status:ready / review:pending. Residual gate: GitHub still reports the old reviewDecision=CHANGES_REQUESTED, so this still needs human re-review/cleanup before merge.

bingran-you · 2026-07-08T08:41:01Z

Daily scan follow-up (2026-07-08): confirmed the previous thermo-nuclear structural blocker has been addressed on current head c4a73d637798d5ba7abda0216ff2d14d3bbd0d00.

The oversized tests/test_init_cli.py file has been split; current line counts are below the 1k threshold (tests/test_init_cli.py 393 lines, tests/test_init_cli_interactive.py 474 lines, tests/test_init_cli_catalog_agents.py 195 lines). CI is green, including test, pip-audit, manifest-parity, and integration-light / rollout-smoke.

Leaving labels as status:ready + review:pending; GitHub still reports reviewDecision=CHANGES_REQUESTED, so this needs reviewer re-review/clearance before merge.

bingran-you · 2026-07-08T12:25:24Z

Users Simulation automation review (2026-07-08): simulation ready, waiting on human re-review on head c4a73d637798d5ba7abda0216ff2d14d3bbd0d00.

Functional evidence:

bench --help, bench init --help, and bench doctor --help passed; init and doctor are visible under Core.
Temp-home user simulation passed: bench init --model openai/gpt-5.4-mini --agent pi-acp --dataset skillsbench@1.1 --sandbox docker --skip-smoke read the local .env, saved config under the temp home, and printed a runnable bench eval run ... command.
bench doctor reached Docker/provider/LiteLLM/model-ping checks; the only failure in the temp simulation was the intentionally dummy API key.
Focused onboarding/init tests passed (136 passed), including tests/test_onboarding.py -k azure_openai_ping_uses_api_key_header_and_gpt5_token_field, which verifies GPT-5 Azure ping uses max_completion_tokens and does not send unsupported max_tokens.
Shared base sweep passed: ruff check ., ty check src, and 149 CLI/SDK/task/usage/skill tests.
Current-head integration-light-jobs artifact passed benchflow-experiment-review: ACP + LLM trajectories and training-ready results.jsonl; reward 1.0, 427,181 tokens, 18 tool calls, total timing 335.8s.

Thermo-nuclear review: no current functional blocker. There is still maintainability pressure from large onboarding surfaces (onboarding.py, init_cmd.py, and test modules), but the prior >1k single-test-file issue has been decomposed. I would keep decomposition as follow-up, not as the current simulation blocker.

Metadata caveat: GLM artifact token/timing coverage is healthy, but result.json.agent_result.cost_usd / price_source are null. This is a shared GLM user-endpoint cost-accounting gap, not an onboarding regression.

Labels: status:ready; changed review label to review:changes-requested because GitHub still reports reviewDecision=CHANGES_REQUESTED. Human re-review is still required before merge.

bingran-you · 2026-07-09T12:11:54Z

Users Simulation automation review (2026-07-09): blocked on current head c4a73d637798d5ba7abda0216ff2d14d3bbd0d00.

The OpenAI-compatible GPT-5 request-shape fix is directionally correct, but live Azure onboarding still fails for realistic users:

In a clean bench init / bench doctor temp-home simulation, the wizard persisted only the auth key into ~/.benchflow/.env. The saved setup path did not reuse the canonical env normalization that derives AZURE_RESOURCE from AZURE_API_ENDPOINT, so the smoke can die on missing AZURE_RESOURCE before the model ping completes.
After supplying the derived Azure resource manually, the GPT-5 ping still failed with max_completion_tokens: 1. Raising the same live ping budget to 8 made the request pass, so the remaining failure is the hard-coded 1-token smoke budget, not the max_completion_tokens field shape.

Passing evidence from the same PR-scoped simulation: CLI help surfaces passed (bench --help, bench init --help, bench doctor --help, bench tasks list-sources, bench tasks check --help), and focused checks passed (136 focused pytest cases, ruff, ty, ruff format --check, git diff --check). No rollout artifacts were generated for this onboarding-only path.

Please route the wizard/doctor saved-env path through the same Azure normalization used by the run path, and make the GPT-5/Azure smoke budget large enough for the provider to return a valid ping response. Moving back to status:blocked / keeping review:changes-requested until that live onboarding path is fixed.

bingran-you · 2026-07-10T16:50:44Z

Daily scan follow-up (2026-07-10): pushed 06071e726 to fix the current-head CI failure on a0e7c474.

Root cause: tests/test_init_cli.py can leave ANTHROPIC_API_KEY in the process environment through the CLI autoload path. The subscription-doctor test was intended to cover a host-subscription setup with no API key, but it did not clear that inherited key, so it failed in full-suite order while passing standalone.

Patch: tests/test_onboarding.py::TestSubscriptionAwareDoctor::test_subscription_login_skips_key_route_and_ping_rows now explicitly clears ANTHROPIC_API_KEY before asserting the subscription skip row.

Local validation in /tmp/benchflow-daily-pr883:

uv run --extra dev python -m pytest tests/test_init_cli.py tests/test_onboarding.py::TestSubscriptionAwareDoctor::test_subscription_login_skips_key_route_and_ping_rows -q -> 19 passed
uv run --extra dev python -m pytest tests/test_init_cli.py tests/test_init_cli_catalog_agents.py tests/test_init_cli_interactive.py tests/test_onboarding.py -q -> 102 passed
uv run --extra dev ruff check tests/test_onboarding.py -> passed
uv run --extra dev ruff format --check tests/test_onboarding.py -> passed
uv run --extra dev ty check src/benchflow/onboarding.py -> passed

Keeping status:blocked / review:changes-requested until the refreshed GitHub CI is green and the human review gate is cleared.

bingran-you · 2026-07-10T17:00:17Z

Users Simulation automation review (2026-07-10): still blocked on current head 06071e726d8ebf59124943816ceb658659bd4522.

The onboarding flow/tests are in much better shape, but the bench doctor sandbox check still has a correctness gap:

benchflow.onboarding.run_doctor(..., sandbox="docker") only checks shutil.which("docker").
The actual Docker sandbox backend already has DockerSandbox.preflight(), which checks both the binary and daemon via docker info.

As written, bench doctor can report a green Docker row when the Docker CLI exists but the daemon is down. Please dispatch the doctor sandbox row through the canonical sandbox backend preflight (or an equivalent shared helper) instead of maintaining a weaker ad hoc probe.

Keeping status:blocked / review:changes-requested.

bingran-you · 2026-07-10T17:00:59Z

Users Simulation automation follow-up (2026-07-10): ready by simulation on current head 06071e726d8ebf59124943816ceb658659bd4522.

The Azure onboarding blocker from #883 (comment) is addressed across the latest commits:

a0e7c4746 routes saved init/doctor env through the run-path env normalization, so Azure endpoint-only setup derives/persists AZURE_RESOURCE before route and ping checks.
GPT-5-family Azure/OpenAI-compatible smoke pings now use max_completion_tokens: 8 instead of the too-small 1-token budget.
06071e726 isolates the subscription-auth doctor test from process ANTHROPIC_API_KEY, preserving the intended “subscription login, no explicit key” coverage after full-suite CI exposed env leakage.

Validation:

Local focused checks on the pushed head passed: uv run pytest tests/test_onboarding.py tests/test_init_cli.py -q -> 76 passed; uv run pytest tests/test_resolve_env_helpers.py tests/test_litellm_config.py tests/test_providers.py -q -> 181 passed; uv run ruff check .; uv run ruff format --check ...; uv run ty check src/; git diff --check.
GitHub CI is green on 06071e726: test, pip-audit, manifest-parity, detect-scope, and integration-light / rollout-smoke.
The integration-light-jobs artifact validates healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, results.jsonl, reward present, 329490 tokens, 17 tool calls, and training_ready=1.

Moving the PR back to status:ready; keeping review:changes-requested because GitHub still reports reviewDecision=CHANGES_REQUESTED and a human re-review is required before merge.

bingran-you · 2026-07-10T17:35:20Z

Users Simulation automation follow-up (2026-07-10): ready by simulation on current head a75d65bdf8d48bba884f6353dae94426ce11caa5.

The fresh doctor sandbox blocker from #883 (comment) is addressed by a75d65bdf:

benchflow.onboarding.run_doctor(..., sandbox="docker") now delegates to DockerSandbox.preflight() instead of only checking shutil.which("docker").
Docker preflight failures are converted into a red doctor row instead of a green false positive.
Added PR feat(cli): bench init onboarding wizard + bench doctor #883 regressions for both the canonical preflight success path and daemon-down failure path.

Validation:

uv run pytest tests/test_onboarding.py tests/test_init_cli.py -q -> 78 passed
focused ruff check / ruff format --check on touched onboarding files -> passed
uv run ty check src/benchflow/onboarding.py -> passed
git diff --check -> passed
GitHub CI is green on a75d65bdf: test, pip-audit, manifest-parity, detect-scope, and integration-light / rollout-smoke.
The latest integration-light-jobs artifact validates healthy with benchflow-experiment-review: ACP trajectory, LLM trajectory, results.jsonl, reward present, 258615 tokens, 14 tool calls, and training_ready=1.

Moving the PR back to status:ready; keeping review:changes-requested because GitHub still reports the prior CHANGES_REQUESTED review and human re-review is required before merge.

symphony-bot added 3 commits July 2, 2026 22:23

Yiminnn temporarily deployed to pypi-internal-preview July 2, 2026 23:00 — with GitHub Actions Inactive

Yiminnn temporarily deployed to pypi-internal-preview July 2, 2026 23:18 — with GitHub Actions Inactive

Yiminnn temporarily deployed to pypi-internal-preview July 3, 2026 05:59 — with GitHub Actions Inactive

Yiminnn temporarily deployed to pypi-internal-preview July 3, 2026 06:06 — with GitHub Actions Inactive

Yiminnn temporarily deployed to pypi-internal-preview July 3, 2026 06:17 — with GitHub Actions Inactive

Yiminnn temporarily deployed to pypi-internal-preview July 3, 2026 06:27 — with GitHub Actions Inactive

Yiminnn temporarily deployed to pypi-internal-preview July 3, 2026 06:42 — with GitHub Actions Inactive

fix(init): type-safe auth-menu branches + confirm line for the enviro…

5489f75

…nment pick ty flagged use[1] (str | None) flowing into str-typed sinks — the ./.env branch now binds a narrowed value; the environment pick also gets its ✓ confirmation line (it was silently branch-less).

Yiminnn temporarily deployed to pypi-internal-preview July 3, 2026 06:52 — with GitHub Actions Inactive

bingran-you requested changes Jul 3, 2026

View reviewed changes

bingran-you added P1 Important debt — must fix soon, but does not block the current release. and removed P2 Anti-pattern / type safety / docs precision / minor schema drift / non-deterministic but contained. review:pending PR is ready-for-review, no reviewer engagement yet. labels Jul 3, 2026

bingran-you added the status:blocked Waiting on external dependency. Add a comment explaining why. label Jul 3, 2026

fix(onboarding): support GPT-5 ping shape

c4a73d6

bingran-you temporarily deployed to pypi-internal-preview July 7, 2026 13:11 — with GitHub Actions Inactive

bingran-you added review:changes-requested Author needs to push more commits before this can merge. and removed review:pending PR is ready-for-review, no reviewer engagement yet. labels Jul 8, 2026

bingran-you added status:blocked Waiting on external dependency. Add a comment explaining why. and removed status:ready Triaged, unassigned, available to claim. labels Jul 9, 2026

fix(onboarding): normalize azure setup env

a0e7c47

bingran-you temporarily deployed to pypi-internal-preview July 10, 2026 16:20 — with GitHub Actions Inactive

test(onboarding): isolate subscription doctor env

06071e7

bingran-you temporarily deployed to pypi-internal-preview July 10, 2026 16:50 — with GitHub Actions Inactive

fix(onboarding): use docker preflight in doctor

a75d65b

bingran-you temporarily deployed to pypi-internal-preview July 10, 2026 17:25 — with GitHub Actions Inactive

bingran-you added status:ready Triaged, unassigned, available to claim. and removed status:blocked Waiting on external dependency. Add a comment explaining why. labels Jul 10, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(cli): bench init onboarding wizard + bench doctor#883

feat(cli): bench init onboarding wizard + bench doctor#883
Yiminnn wants to merge 28 commits into
mainfrom
feat/bench-init

Yiminnn commented Jul 2, 2026

Uh oh!

Yiminnn commented Jul 3, 2026

Uh oh!

bingran-you left a comment

Uh oh!

bingran-you commented Jul 3, 2026

Uh oh!

bingran-you commented Jul 7, 2026

Uh oh!

bingran-you commented Jul 8, 2026

Uh oh!

bingran-you commented Jul 8, 2026

Uh oh!

bingran-you commented Jul 9, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Yiminnn commented Jul 2, 2026

bench init

bench doctor

Quality

Uh oh!

Yiminnn commented Jul 3, 2026

UX redesign (commit 1fbfb59): choose, don't type

Uh oh!

bingran-you left a comment

Choose a reason for hiding this comment

Uh oh!

bingran-you commented Jul 3, 2026

Uh oh!

bingran-you commented Jul 7, 2026

Uh oh!

bingran-you commented Jul 8, 2026

Uh oh!

bingran-you commented Jul 8, 2026

Uh oh!

bingran-you commented Jul 9, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

bingran-you commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`bench init`

`bench doctor`

UX redesign (commit `1fbfb59`): choose, don't type