Repository navigation
Add logging - #127
Open
ev-br wants to merge 12 commits into
Open
Add logging#127ev-br wants to merge 12 commits into
ev-br wants to merge 12 commits into
Conversation
ev-br
force-pushed
the
logging-phases-1-2
branch
from
November 20, 2026 12:25
dc0ff95 to
25ab8fa
Compare
…the plumbing) bx bench and bx compare now write bx.log in the run's output folder: a header with the command as given, the config found, each setting with where it came from, and a footer with the outcome, exit code and elapsed time. The terminal is unchanged at the default verbosity: what bx prints to stderr goes through the logger as console messages, and a record that is not one is shown only with -v or -vv. - new benchx/log.py: the console adapter and handler, the file and --log handlers, one line format (UTC to the millisecond, pid, logger, level; continuation lines indented), the footer, log.once, log.degraded and log.command - the switch: --log, --no-log, --log-file PATH (one destination, mutually exclusive), `logging` and `log_level` in benchx.conf.json, $BENCHX_LOG_LEVEL and --log-level, $BENCHX_VERBOSE with -v and -q; a command without an output folder logs only when a flag asks; an unusable log file is refused before anything runs - cli.main wraps the command: a crash is a console error with its traceback and exit status 1, SystemExit passes through, an interrupt or SIGTERM leaves a footer - the run key and the output folder are settled before any target is resolved, through one session.run_directory(); the harness flag check moves into the CLI's check phase, so a refusal for a flag still leaves no folder, while a refusal inside the session leaves a folder holding only bx.log, which says why - Config.pick_with_source and the env argument, Config.log_file and Config.own_paths; bx config show gets logging, log level and verbosity rows - adapters/asv.py: its three stderr prints are console messages Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
docs/benchx.md: a paragraph on the log (where it is, its header and footer, the flags, the environment variables and the config keys, -v and -q), saying plainly that a bx.log records the command line as given and can contain secrets. docs/design/prototype-design.md: the output channels, and the logging handlers beside them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lan, phase 2) The log now follows a run. Per order: the protocol, what it records (by name, never a value of the inherited environment), where each environment name came from, how the subject's identity was found, the interpreter, and a dirty tree (INFO the first time in a command, DEBUG after, with the first ten paths git lists). Per case: `starting` before it runs, then one line by its result status (`error (harness.error): ValueError: boom, exit 0, 1.2 s, load 0.42`, a WARNING; a non-zero exit is a WARNING too), a files line with absolute paths, and the harness's stdout and stderr (the last 40 lines, DEBUG, or WARNING for a case that did badly). At the end of an order: the results by status. - runner: refusals through one _refuse(); a rerun of a run key says its result files are overwritten; a case loop that stops partway says how many cases were written and that bx ingest stores them; an unknown dirty state is a WARNING naming the cause - session: the run key and folder, each side, each slot before and after it (with the store's outcome), the variants' candidate parameters and per-value filters, the baseline's origin, and what the store rejected; a round that stops after an earlier slot was ingested says how to compare what is there - collectors: command, exit, elapsed, a progress line each, the reason for a failure, up to 40 differences - adapters: run_bounded logs the command, how it ended and its output (a listing's stdout only when it failed); run_case logs its command and a missing or unparseable native output; list_cases logs the cases; target logs the provider call and the sidecar; bench logs the shorthand's tree - snapshot: failing git commands, why a tree is dirty, why its state is unknown - cli: the footer says what an exit-0 command left undone (`1 of 4 cases not measured`); ingest records - log: harness_output(), and a small buffer so that records made in a command's own checks reach the log once it starts Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ging plan, phase 3) - compare(): a thin wrapper over _compare() that logs the inputs and the outcome from the document it returns, so every return path is covered, and the document is unchanged: one record per unit with its verdict trail (rounds, reference, effect, noise, k x noise, reason, attempts), one per failed invariant (ERROR when it is about the run), the excluded results grouped by reason, the missing units, and a summary that is a WARNING when anything was indeterminate, excluded or failed. The decisions only _compare() makes are logged where they happen: how --baseline matched, a second success of a side in one round overwriting the first, rounds dropped for a success on one side only, successes with no observations, and min_rounds below 2 - a read-mode comparison appends its records to the run folder's bx.log when the folder exists, and is skipped silently when it cannot be written; --results logs how many documents it ingested, and a rejection says the comparison was not run - snapshot: a working tree identified by a hash of its files (INFO, WARNING when git failed), an unreadable file in it, the exclusions behind a tree id, the CMake configuration (WARNING when there is no cache or no compiler file), a setup file that is missing or unreadable (naming the variable, never its value), the CPU model, the load average and unreadable system facts; the refusal for a checkout that cannot be located says why - store: what a sweep found, every error of a rejected document at DEBUG, an idempotency-conflict with both digests, work orders that no result references, a new quantity's unit, the schema promotion by field path and type, the time of each read and write, and an absent file read as an empty store - adapters: pyperf's functions left out without a filter, those with no version, those whose params could not be read, a name-only filter match; gbench's partial case; asv's conf that is not JSON - session, target, cli: how a side was found, the source URI and where it came from, the shorthand's mode, the project at INFO, a provider's SOURCE_REF default Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s for R11 and R22 - store: a write that fails (disk full, permissions) is an ERROR naming the store, the rows that were to be added and the partial temporary file; so is a batch that cannot be converted to rows - session: delivery that fails says where the results are and that bx ingest stores them - tests: the logger passes only the levels somebody wants (R11); a failed case's `not measured` section and `bx compare --tables`, with and without --all, keep stdout free of log lines under --log (R22) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…es, the context, history and validate - compare: which result supplied each side's declarations, a WARNING naming the fields when the side's other results disagree (never their values), and the facts that differ between the sides but that the identity policy ignores - runner: a Google Benchmark context that says CPU frequency scaling is on, or the library is a debug build, is a WARNING once per command - snapshot.ignore: the paths now left out of every tree identity - cli: `bx history` says when no series matches and warns when a prefix matches several, `bx validate` logs its kind and each valid file Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…warning reasons, tables, validation - asv: each setting of a case's plan with where it came from (the order, sample_time, warmup_time, timeout, or the default), the order's min_time when the fixed loops make it useless, a `repeat` that is not a form it reads; a `skipped.json`, `facts.json` or worker-environment file that exists and cannot be read; a failed module's traceback and the benchmarks the suite switched off with skip_benchmark, which asv_runtime now returns in its listing (the worker logs nothing) - the case line gives the reason behind a quality warning (env-mismatch: how many workers matched; env-capture-missing: missing or unparseable; setup-per-call-unbatched: median and minimum) and a partial case's completed over attempted; the facts a harness replaces in the context's are named - tables: what --tables showed and hid, the documents it skipped, and the units the side tables hold that the comparison dropped - core.validation: every schema error at DEBUG with its path and keyword; session: the causes behind the conversions that replace them; `bx series` and `bx head` say when nothing matches; an inferred --vary says how many results it scanned - fix: the setup-per-call detail read an observation as a number, and observations may be dicts Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… consults Every command now logs, at DEBUG, the arguments as argparse understood them (beside the command line as given, which the header already echoes) and the environment variables benchx reads that decide what it does: BENCHX_HOME, BENCHX_VERBOSE, BENCHX_LOG_LEVEL, BENCHX_SETUP_FILE, BENCHX_ENV_ALLOWLIST and BENCHX_SCHEMAS, each with its value or `not set`, and whether PATH is set. These are settings someone typed for benchx, a fixed list; the rest of the inherited environment is still never dumped. The docs say so. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Right after a case's `starting` record, at DEBUG: its workload and its parameter values (a parametrized benchmark's own, as asv's repr() strings, and the order's -e variables, which the runner records as workload parameters too). The listing says once, per parametrized benchmark, its parameters, their values and how many combinations there are. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t says An integer, 0 by default (nothing more than before). Set by `--harness-verbosity N`, else by each `-v` past `-vv` (`-vvv` is `-vv` and 1, `-vvvv` is `-vv` and 2), else `$BENCHX_HARNESS_VERBOSITY`, else `adapter.verbosity` in the config file. pyperf and asv are given `--verbose` when it is 1 or more, Google Benchmark `--v=N`. - it is an optional work-order field, `harness_verbosity`, so the order records what ran and `bx run` repeats it; it is left out of the series identity, and each result's provenance.info records it - the log keeps up to 2000 lines of each stream of a case's output when it is on, 40 otherwise - `$BENCHX_VERBOSE` accepts values above 2 for the same rule; `bx config show` has a row - docs: benchx.md, the work-order schema's table and runner-schema.md Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…a call starts with a line `bx bench` and `bx run` printed their table, the cases that were not measured, the result line and the ingest counts with `print`, outside logging. So the log lacked a round's numbers, and `-q` and `-qq` could not quiet them (a failed case exits 0 and showed only there). - `log.console().out(text, tier)`: a record marked `console=True, stdout=True`, one record a line. A stdout handler writes it unless `-q` or `-qq` hides it by its tier (the table 0, the result line 1, the `not measured` section never); the stderr console skips it, `-v` included; the log keeps it at every verbosity. A failed write to stdout (a closed pipe) is not swallowed. - `-q` leaves the `not measured` section and the result line, `-qq` only the section; a run that measured everything is silent at `-qq`. When the table is hidden and says nothing of a failure, `N of M cases not measured` stands in. `bx ingest`, `bx compare`, the queries and `--json` still print as before. - A call starts with a progress line, `measuring SUITE on TARGET (round R/N, slot S)` for `bx bench` and `running the order FILE` for `bx run`, which `-q` keeps and `-qq` does not (`console().progress()`: a record's tier lets it past the level). It is new on the default stderr. - The ingest counts after a run are an ordinary console message on stderr, so `-q` hides them; `bx ingest` prints its counts on stdout as its answer. - `--round R/N` says the session has N rounds, numbered 0 to N-1; only the start line uses N, and the work order records R alone. `--round R` works as before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…und R/N Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ev-br
force-pushed
the
logging-phases-1-2
branch
from
November 20, 2026 15:10
ebf54dc to
f871007
Compare
Collaborator
Author
|
As a specific example, in benchx-dev/benchx_examples#18 applying the diff below to the "workbench" script makes the loop over the rounds mostly quiet and the final diff --git a/examples/numpy-revisions/compare.sh b/examples/numpy-revisions/compare.sh
index 0b7e643..7992bd3 100755
--- a/examples/numpy-revisions/compare.sh
+++ b/examples/numpy-revisions/compare.sh
@@ -57,7 +57,7 @@ measure() {
export BUILD PYTHONPATH
python -c 'import numpy; print(numpy.__version__, numpy.__file__)' # which numpy this side measures
bx bench @numpy --adapter asv --suite "$NUMPY/benchmarks/benchmarks" --filter "$filter" \
- --run-key "$run_key" --round "$round" --slot "$slot"
+ --run-key "$run_key" --round "$round" --slot "$slot" -q
}
for round in 0 1 2 3 4; do
@@ -65,4 +65,4 @@ for round in 0 1 2 3 4; do
measure "$contender" "$contender_build" "$round" $((2 * round + 1))
done
-bx compare --run "$run_key" --baseline "$baseline" --tables
+bx compare --run "$run_key" --baseline "$baseline" --tables --all |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes gh-85
Add basic logging to
benchx. The goal is to make it easier to trace what benchx does under the hood, and make it easier to control the verbosity of the terminal output (cf #117) without loosing the ability to trace the details (what benchx is actually doing, where do various parameters come from etc etc).Two examples of where it's helpful:
--compareoutput needs improvement #117 (comment).pyperfwarnings about benchmark stability end up in the log (cf low reliability of no-op code change benchmarks with--comparewith asv adaptor #126)All output is routed through the logger. There are two handlers: file and console. By default, console gets messages INFO level and above, the file in
out/getsDEBUGlevel and above.Logging can be controlled via CLI or
benchx.conf.jsonor environment variables, with a usual precedence:--log-level LEVELdebug(default),info,warningorerror, in any case$BENCHX_LOG_LEVELlog_levelinbenchx.conf.jsonThe order is flag, then environment, then file, then
debug. A bad environment value is refused (BENCHX_LOG_LEVEL=loud: not one of …).--no-log --log-level …is a usage error, since there is nothing to set a level on.What each level keeps
fallbacks, and the footer's "left undone" phrases (
1 of 4 cases not measured).result line, the order summary, and each unit's verdict.
lines, workload parameters, file paths, harness output, git commands, and store reads.
--log-level warningdrops the start and end records, because they are INFO.Where the log goes
{out}/bx.login the run's output folder, forbench,compareandrun --out. Other commands lognothing unless asked.
--no-logturns it off,--logsends it to stderr,--log-file PATHappends to a file ({out}and{root}work in the path). The config keyloggingtakesfalse,trueor a path. Flags beat the file.CLI verbosity controls
Additionally,
bx benchandbx compareget new CLI arguments,-v/-qto control verbosity of the terminal output.These options only change what
bxwrites to stderr.bxprinted before logging existed: progress, refusals, rejections-q-qq-v-vv-vvv,-vvvv, …-vv, and each extravalso asks the harness to say more:-vvvis--harness-verbosity 1,-vvvvis--harness-verbosity 2-vand-qadd up, so-v -qis the default.$BENCHX_VERBOSEsets the net count (-2 or more) when noflag is given.
-vand-qnever change it. Its content is set by--log-level(defaultdebug).--log:-vand-qare ignored, because stderr is then the log itself.--harness-verbosity Nbeats the count from-v.Harness verbosity
Harnesses themselves have contollable verbosity levels. Cranking verosity from the harness can be useful for debugging / performance tuning. Thus, add a new
--harness-verbosity Noption, which is forwarded to harnesses. The result can be very verbose, so it is off by default.The verbosity level is added to
provenance.info.harness_verbosity, so that replaying work orders viabx rungives identical results.The patch size looks scary but the vast majority of the diff is fairly trivial plumbing.