Skip to content

Make "dirty" determination ignore untracked files - #130

Draft
ev-br wants to merge 14 commits into
benchx-dev:mainfrom
ev-br:dirty-tracked-files-only
Draft

ev-br wants to merge 14 commits into
benchx-dev:mainfrom
ev-br:dirty-tracked-files-only

Conversation

@ev-br

@ev-br ev-br commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

(on top of #127, will need a rebase; only two last commits in this PR are relevant, one for the docs, one for the code)

In #117 , one of the discussion points is what exactly makes a tree "dirty".

Currently, we look at all files, and having an untracked file in the git-tracked directory makes the whole tree dirty.
Ralf argued that that this is too aggressive, and the rule should be that only changed to tracked files count.

So this PR is the middle ground, per the gh-117 discussion:

  • untracked files are ignored, so that a checkout + untracked files is "clean";
  • when untracked files are present, the dirty/clean check emits a log message naming the files.

ev-br's Claude Code Agent and others added 14 commits November 20, 2026 14:58
…the plumbing)

bx bench and bx compare now write bx.log in the run's output folder: a header with
the command as given, the config found, each setting with where it came from, and a
footer with the outcome, exit code and elapsed time. The terminal is unchanged at the
default verbosity: what bx prints to stderr goes through the logger as console messages,
and a record that is not one is shown only with -v or -vv.

- new benchx/log.py: the console adapter and handler, the file and --log handlers, one
  line format (UTC to the millisecond, pid, logger, level; continuation lines indented),
  the footer, log.once, log.degraded and log.command
- the switch: --log, --no-log, --log-file PATH (one destination, mutually exclusive),
  `logging` and `log_level` in benchx.conf.json, $BENCHX_LOG_LEVEL and --log-level,
  $BENCHX_VERBOSE with -v and -q; a command without an output folder logs only when a
  flag asks; an unusable log file is refused before anything runs
- cli.main wraps the command: a crash is a console error with its traceback and exit
  status 1, SystemExit passes through, an interrupt or SIGTERM leaves a footer
- the run key and the output folder are settled before any target is resolved, through one
  session.run_directory(); the harness flag check moves into the CLI's check phase, so a
  refusal for a flag still leaves no folder, while a refusal inside the session leaves a
  folder holding only bx.log, which says why
- Config.pick_with_source and the env argument, Config.log_file and Config.own_paths;
  bx config show gets logging, log level and verbosity rows
- adapters/asv.py: its three stderr prints are console messages

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
docs/benchx.md: a paragraph on the log (where it is, its header and footer, the flags, the
environment variables and the config keys, -v and -q), saying plainly that a bx.log records
the command line as given and can contain secrets. docs/design/prototype-design.md: the
output channels, and the logging handlers beside them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lan, phase 2)

The log now follows a run. Per order: the protocol, what it records (by name, never a value of
the inherited environment), where each environment name came from, how the subject's identity was
found, the interpreter, and a dirty tree (INFO the first time in a command, DEBUG after, with the
first ten paths git lists). Per case: `starting` before it runs, then one line by its result status
(`error (harness.error): ValueError: boom, exit 0, 1.2 s, load 0.42`, a WARNING; a non-zero exit is a
WARNING too), a files line with absolute paths, and the harness's stdout and stderr (the last 40
lines, DEBUG, or WARNING for a case that did badly). At the end of an order: the results by status.

- runner: refusals through one _refuse(); a rerun of a run key says its result files are overwritten;
  a case loop that stops partway says how many cases were written and that bx ingest stores them;
  an unknown dirty state is a WARNING naming the cause
- session: the run key and folder, each side, each slot before and after it (with the store's outcome),
  the variants' candidate parameters and per-value filters, the baseline's origin, and what the store
  rejected; a round that stops after an earlier slot was ingested says how to compare what is there
- collectors: command, exit, elapsed, a progress line each, the reason for a failure, up to 40 differences
- adapters: run_bounded logs the command, how it ended and its output (a listing's stdout only when it
  failed); run_case logs its command and a missing or unparseable native output; list_cases logs the
  cases; target logs the provider call and the sidecar; bench logs the shorthand's tree
- snapshot: failing git commands, why a tree is dirty, why its state is unknown
- cli: the footer says what an exit-0 command left undone (`1 of 4 cases not measured`); ingest records
- log: harness_output(), and a small buffer so that records made in a command's own checks reach the
  log once it starts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ging plan, phase 3)

- compare(): a thin wrapper over _compare() that logs the inputs and the outcome from the
  document it returns, so every return path is covered, and the document is unchanged: one record
  per unit with its verdict trail (rounds, reference, effect, noise, k x noise, reason, attempts),
  one per failed invariant (ERROR when it is about the run), the excluded results grouped by reason,
  the missing units, and a summary that is a WARNING when anything was indeterminate, excluded or
  failed. The decisions only _compare() makes are logged where they happen: how --baseline matched, a
  second success of a side in one round overwriting the first, rounds dropped for a success on one side
  only, successes with no observations, and min_rounds below 2
- a read-mode comparison appends its records to the run folder's bx.log when the folder exists, and
  is skipped silently when it cannot be written; --results logs how many documents it ingested, and a
  rejection says the comparison was not run
- snapshot: a working tree identified by a hash of its files (INFO, WARNING when git failed), an
  unreadable file in it, the exclusions behind a tree id, the CMake configuration (WARNING when there is
  no cache or no compiler file), a setup file that is missing or unreadable (naming the variable, never
  its value), the CPU model, the load average and unreadable system facts; the refusal for a checkout that
  cannot be located says why
- store: what a sweep found, every error of a rejected document at DEBUG, an idempotency-conflict with
  both digests, work orders that no result references, a new quantity's unit, the schema promotion by
  field path and type, the time of each read and write, and an absent file read as an empty store
- adapters: pyperf's functions left out without a filter, those with no version, those whose params
  could not be read, a name-only filter match; gbench's partial case; asv's conf that is not JSON
- session, target, cli: how a side was found, the source URI and where it came from, the shorthand's mode,
  the project at INFO, a provider's SOURCE_REF default

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s for R11 and R22

- store: a write that fails (disk full, permissions) is an ERROR naming the store, the rows that were
  to be added and the partial temporary file; so is a batch that cannot be converted to rows
- session: delivery that fails says where the results are and that bx ingest stores them
- tests: the logger passes only the levels somebody wants (R11); a failed case's `not measured` section
  and `bx compare --tables`, with and without --all, keep stdout free of log lines under --log (R22)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…es, the context, history and validate

- compare: which result supplied each side's declarations, a WARNING naming the fields when the side's
  other results disagree (never their values), and the facts that differ between the sides but that the
  identity policy ignores
- runner: a Google Benchmark context that says CPU frequency scaling is on, or the library is a debug
  build, is a WARNING once per command
- snapshot.ignore: the paths now left out of every tree identity
- cli: `bx history` says when no series matches and warns when a prefix matches several, `bx validate`
  logs its kind and each valid file

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…warning reasons, tables, validation

- asv: each setting of a case's plan with where it came from (the order, sample_time, warmup_time,
  timeout, or the default), the order's min_time when the fixed loops make it useless, a `repeat`
  that is not a form it reads; a `skipped.json`, `facts.json` or worker-environment file that exists
  and cannot be read; a failed module's traceback and the benchmarks the suite switched off with
  skip_benchmark, which asv_runtime now returns in its listing (the worker logs nothing)
- the case line gives the reason behind a quality warning (env-mismatch: how many workers matched;
  env-capture-missing: missing or unparseable; setup-per-call-unbatched: median and minimum) and a
  partial case's completed over attempted; the facts a harness replaces in the context's are named
- tables: what --tables showed and hid, the documents it skipped, and the units the side tables hold that
  the comparison dropped
- core.validation: every schema error at DEBUG with its path and keyword; session: the causes behind the
  conversions that replace them; `bx series` and `bx head` say when nothing matches; an inferred --vary
  says how many results it scanned
- fix: the setup-per-call detail read an observation as a number, and observations may be dicts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… consults

Every command now logs, at DEBUG, the arguments as argparse understood them (beside the command line as
given, which the header already echoes) and the environment variables benchx reads that decide what it
does: BENCHX_HOME, BENCHX_VERBOSE, BENCHX_LOG_LEVEL, BENCHX_SETUP_FILE, BENCHX_ENV_ALLOWLIST and
BENCHX_SCHEMAS, each with its value or `not set`, and whether PATH is set. These are settings someone
typed for benchx, a fixed list; the rest of the inherited environment is still never dumped. The docs say so.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Right after a case's `starting` record, at DEBUG: its workload and its parameter values (a
parametrized benchmark's own, as asv's repr() strings, and the order's -e variables, which the runner
records as workload parameters too). The listing says once, per parametrized benchmark, its parameters,
their values and how many combinations there are.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t says

An integer, 0 by default (nothing more than before). Set by `--harness-verbosity N`, else by each `-v`
past `-vv` (`-vvv` is `-vv` and 1, `-vvvv` is `-vv` and 2), else `$BENCHX_HARNESS_VERBOSITY`, else
`adapter.verbosity` in the config file. pyperf and asv are given `--verbose` when it is 1 or more, Google
Benchmark `--v=N`.

- it is an optional work-order field, `harness_verbosity`, so the order records what ran and `bx run`
  repeats it; it is left out of the series identity, and each result's provenance.info records it
- the log keeps up to 2000 lines of each stream of a case's output when it is on, 40 otherwise
- `$BENCHX_VERBOSE` accepts values above 2 for the same rule; `bx config show` has a row
- docs: benchx.md, the work-order schema's table and runner-schema.md

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…a call starts with a line

`bx bench` and `bx run` printed their table, the cases that were not measured,
the result line and the ingest counts with `print`, outside logging. So the log
lacked a round's numbers, and `-q` and `-qq` could not quiet them (a failed case
exits 0 and showed only there).

- `log.console().out(text, tier)`: a record marked `console=True, stdout=True`,
  one record a line. A stdout handler writes it unless `-q` or `-qq` hides it by
  its tier (the table 0, the result line 1, the `not measured` section never);
  the stderr console skips it, `-v` included; the log keeps it at every
  verbosity. A failed write to stdout (a closed pipe) is not swallowed.
- `-q` leaves the `not measured` section and the result line, `-qq` only the
  section; a run that measured everything is silent at `-qq`. When the table is
  hidden and says nothing of a failure, `N of M cases not measured` stands in.
  `bx ingest`, `bx compare`, the queries and `--json` still print as before.
- A call starts with a progress line, `measuring SUITE on TARGET (round R/N,
  slot S)` for `bx bench` and `running the order FILE` for `bx run`, which `-q`
  keeps and `-qq` does not (`console().progress()`: a record's tier lets it past
  the level). It is new on the default stderr.
- The ingest counts after a run are an ordinary console message on stderr, so
  `-q` hides them; `bx ingest` prints its counts on stdout as its answer.
- `--round R/N` says the session has N rounds, numbered 0 to N-1; only the start
  line uses N, and the work order records R alone. `--round R` works as before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…und R/N

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… files are logged

The dirty flag and the working-tree id covered untracked files too, so nearly
every working checkout was dirty: scratch notes, a results folder, a build
directory, none of which a benchmark reads. A user found it too aggressive and
too verbose; a maintainer found tracked files only probably fine, provided the
log says what is left out.

- The flag is `git status --untracked-files=no`: an edit, a deletion, a rename,
  a mode change, a submodule at another commit or with changes of its own.
- The tree id is HEAD's tree updated with the tracked files as they are now
  (a scratch index: `read-tree HEAD`, `add -u`, `write-tree`), so a clean tree
  has HEAD's own id. Untracked files never change it, and an unreadable
  untracked file no longer makes the state `unknown`.
- `git_identity` lists the untracked paths (`untracked`; `__pycache__`, `out/`
  and benchx's own paths left out), and the runner says them in one INFO record
  per tree and command: the count and the first ten.
- The special case that kept an untracked `out/` out of the identity is gone.
- `_run` no longer strips the first column of a status line: ` M a.py` was
  logged as `M a.py`.

Tests that made a tree dirty with an untracked file now edit a tracked one.
The numpy end-to-end test is changed the same way and was not run here.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
benchmark-result-schema.md §4.1 said tracked and untracked files count in the
working-tree id, and named the asv failure mode, uncommitted changes silently
ignored, as the one to avoid. This amends it: the dirty flag and the tree id
cover tracked files only, the recipe is `read-tree HEAD && add -u &&
write-tree`, and a new paragraph, "Tracked files only", gives the reason and
states the cost: a new file that is not yet added is measured but is not part
of the id, and the result carries no mark of it; the runner's log names the
untracked paths instead. The submodule limit now says the flag still shows
what the id cannot.

harness-adapter.md, prototype-design.md and benchx.md follow.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ev-br
ev-br marked this pull request as draft November 20, 2026 17:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant