feat: add --progress-file for machine-readable case progress events

[JHWang-1997]

Problem / motivation

skill-up run currently exposes case-level progress only through human-readable terminal output.

CI systems that need to periodically report evaluation progress must parse stdout, which is fragile because:

• the output format is intended for humans rather than machines;
• concurrent cases may interleave output;
• verbose logs may contain unrelated messages;
• wording or formatting changes can break downstream parsers.

evaluator.ProgressObserver already receives callbacks when cases start and complete, but these events are only forwarded to the terminal UI.

We need a stable, machine-readable progress stream that CI jobs can consume while an evaluation is still running.

Proposed solution

Add an optional --progress-file flag to skill-up run:

skill-up run --progress-file ./progress.jsonl

When specified, skill-up should:

1. Create or truncate the target file when the run starts.
2. Append one JSON object per line as ProgressObserver callbacks occur.
3. Make each event immediately visible to readers so CI wrappers can tail or periodically read the file.
4. Preserve the existing terminal progress output.
5. Leave current behavior unchanged when the flag is not specified.

Example events:

{"schema_version":"1","event":"case_started","time":"2026-08-18T10:00:00.002Z","iteration":1,"case_index":1,"case_total":2,"case_id":"case-1","configuration":"with_skill","title":"Basic flow"}
{"schema_version":"1","event":"case_completed","time":"2026-08-18T10:01:10.451Z","iteration":1,"case_index":1,"case_total":2,"case_id":"case-1","configuration":"with_skill","status":"PASS","pass_rate":1.0,"duration_ms":70449}

Suggested fields:

Field	Description
schema_version	Event schema version, initially "1"
event	case_started or case_completed
time	Event timestamp in RFC 3339 format
iteration	Current iteration number
case_index	Existing observer case index
case_total	Total number of case tasks in the iteration
case_id	Case identifier
configuration	with_skill or without_skill
title	Case title, for case_started
status	PASS, FAIL, ERROR, or SKIP, for case_completed
pass_rate	Judge pass rate when available
duration_ms	Case duration, for case_completed

The progress writer should be safe when cases run concurrently. Each JSON line must be written atomically under synchronization so readers never observe interleaved or partial events.

Supporting benchmark and multi-iteration runs requires passing configuration, iteration, and duration information through the observer path. In benchmark mode, consumers can distinguish duplicate case IDs using (case_id, configuration).

Example CI usage:

skill-up run --progress-file ./progress.jsonl &
skill_up_pid=$!

while kill -0 "$skill_up_pid" 2>/dev/null; do
  completed=$(jq -s '[.[] | select(.event == "case_completed")] | length' progress.jsonl 2>/dev/null || echo 0)
  echo "Completed cases: ${completed}"
  sleep 10
done

wait "$skill_up_pid"

Error handling

• If the progress file cannot be created or opened, fail at startup with a clear error because the user explicitly requested the output.
• If writing fails after evaluation has started, emit a warning and allow the evaluation to continue with its normal result and exit code.

Acceptance criteria

• skill-up run accepts --progress-file <path>.
• No progress file is created when the flag is absent.
• The file is created or truncated once at run startup.
• Each observer callback appends exactly one newline-terminated JSON event.
• Events are visible while evaluation is still running.
• Every line is independently valid JSON.
• Concurrent case execution cannot produce interleaved or corrupted lines.
• Events distinguish with_skill and without_skill executions.
• Events identify the current iteration.
• Existing terminal progress output remains unchanged.
• Tests cover event serialization, concurrent writes, benchmark mode, multiple iterations, and write failures.
• CLI documentation and changelog are updated.

Alternatives considered

Parse stdout

This works without code changes, but stdout is a human-facing interface and is not a stable machine-readable contract.

Poll report files

Reports are generated after an iteration completes, so they cannot provide case-level progress during execution.

Use OTLP telemetry

OTLP is appropriate for observability platforms, but requiring a collector is unnecessarily heavy for CI jobs that only need to report completed case counts.

Add an HTTP callback

A callback would introduce authentication, retry, timeout, and delivery semantics. A local JSONL stream keeps skill-up simple and lets CI wrappers choose how to forward progress.

Additional context

The existing integration point is evaluator.ProgressObserver, currently implemented by uiProgressObserver in internal/cli/run.go.

The new file observer can be composed with the existing UI observer so both receive the same callbacks.

[zpzjzj]

I support adding a machine-readable event stream, while keeping the existing stdout output unchanged. Bazel's Build Event Protocol (BEP) is a useful model here: stdout remains optimized for humans, while a separate structured stream is optimized for CI systems and other consumers.

I suggest treating this as a lightweight evaluation event log rather than only a progress file:

• Prefer --event-log <path> over --progress-file <path>. Progress is the first use case, but the same stream may later carry lifecycle, summary, diagnostic, and artifact events.
• Use JSONL for v1. There is no need to copy BEP's protobuf encoding or full event DAG yet.
• Include lifecycle events, not only per-case events: run_started, iteration_started, case_started, case_completed, iteration_completed, and run_finished.
• Give every event a monotonically increasing sequence_number and a stable invocation_id. This lets consumers detect ordering, gaps, and which events belong to the same run.
• Model benchmark progress in terms of tasks rather than source cases. A case may expand into multiple configurations, so task_index / task_total is less ambiguous than case_index / case_total.
• Always emit a terminal run_finished event for normal completion, including failed evaluations. Consumers can distinguish a completed run from a truncated stream. As with BEP, an abruptly terminated process may leave the stream without its terminal event.

For example:

{"schema_version":1,"sequence_number":3,"invocation_id":"<uuid>","time":"2026-08-19T10:00:00Z","event":"case_completed","payload":{"task_id":"iteration-1/case-1/with_skill","iteration":1,"case_id":"case-1","configuration":"with_skill","task_index":1,"task_total":4,"completed_tasks":1,"status":"PASS","pass_rate":1.0,"duration_ms":70449}}

And the terminal event could look like:

{"schema_version":1,"sequence_number":10,"invocation_id":"<uuid>","time":"2026-08-19T10:05:00Z","event":"run_finished","last_event":true,"payload":{"status":"SUCCESS","completed_tasks":4,"passed":3,"failed":1,"errored":0,"duration_ms":300000}}

On the implementation side, I would avoid continuing to add positional parameters to ProgressObserver. A more extensible boundary would be something like:

type EventSink interface {
    Publish(context.Context, Event) error
}

The CLI can compose a UI sink with a JSONL file sink, and future consumers could add an OTLP, callback, or socket sink without changing evaluator callbacks again.

For the JSONL writer:

• Open or truncate the file once at startup.
• Assign sequence_number, marshal, and write under the same lock.
• Append a newline and perform one Write per record.
• Do not fsync every event or rely on a delayed buffered flush.
• Define the compatibility contract as: every newline-terminated record is valid JSON, and readers must ignore or buffer an unterminated trailing record. A regular file cannot guarantee that a concurrently reading process will never observe a partial final write, especially on crash or disk failure.

For errors, failing to create/open the requested event log should fail startup. A mid-run write failure should become sticky: the evaluation may continue so reports can still be produced, but the command should return a non-zero exit status at the end. Silently warning and exiting successfully would make CI believe it has a complete event stream when it does not.

This keeps the initial implementation small, while preserving the main BEP design principle: a stable, machine-readable protocol that is separate from human-readable console output.

References: Bazel Build Event Protocol, BEP examples.