Skip to content

Add run_api_docs and enforce public-API docstrings by default in run_qa (2.0.0) - #19

Merged
ChrisRackauckas merged 2 commits into
mainfrom
api-docs-qa-check
Jul 10, 2026
Merged

Add run_api_docs and enforce public-API docstrings by default in run_qa (2.0.0)#19
ChrisRackauckas merged 2 commits into
mainfrom
api-docs-qa-check

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jul 10, 2026

Copy link
Copy Markdown
Member

This draft PR should be ignored until reviewed by @ChrisRackauckas.

Motivation

A wave of near-identical PRs (SciML/SciMLSensitivity.jl#1537, SciML/DiffEqGPU.jl#472) added a hand-copied test/QA/public_api_docs.jl to individual SciML repos, each re-implementing the same check: every exported name has a docstring (and is rendered in the docs). This PR factors that into the shared harness and turns it on by default, so the fleet enforces public-API docstrings from a plain run_qa(MyPkg) and repos delete their bespoke files.

What's added

  • public_api_names(pkg) — the sorted public API: exported names, plus public-declared names on Julia >= 1.11 (exported-only on the 1.10 LTS, so no per-repo if VERSION guards).
  • run_api_docs(pkg; docstrings=true, rendered=false, docs_src, ignore, rendered_ignore, docstrings_broken, rendered_broken):
    • docstrings (on by default): every public name has a docstring. The lookup is Base.Docs.doc on the binding, so a name re-exported from a dependency counts as documented via that dependency — a repo isn't forced to redocument re-exports.
    • rendered (opt-in): every public name appears in a ```@docs block under docs_src (defaults to <pkgroot>/docs/src). An ```@autodocs block satisfies it wholesale.
    • ignore/rendered_ignore for exceptions; docstrings_broken/rendered_broken to mark the check @test_broken for a mid-migration repo (auto-flags an Unexpected Pass once fully documented).
  • run_qa now runs api_docs by default (api_docs = true) via api_docs_kwargs. explicit_imports stays opt-in (its per-repo ignore-lists make default-on impractical).
  • REPL added as a (stdlib) dep, imported for its load-time side effect: on Julia >= 1.11 the Base.Docs.doc(::Binding) methods live in REPL and are absent in a bare Pkg.test process. (The copied per-repo files only worked because the test env happened to load REPL transitively.)

Breaking change -> 2.0.0

Turning api_docs on by default changes the observable behavior of the pre-existing run_qa for every caller: a repo that upgrades and has undocumented public API will see its QA lane go red where it was green. Per SemVer that is a breaking change, so this is a major bump to 2.0.0 (not 1.9.0). Escape hatches: run_qa(MyPkg; api_docs = false), or api_docs_kwargs = (; ignore = (...)) / (; docstrings_broken = true).

Note: this deliberately reverses the opt-in stance explicit_imports uses. If you'd rather treat "stricter QA on a bump" as non-breaking and ship it as 1.9.0 (or keep api_docs opt-in), say so and I'll adjust.

Migration path for repos

A repo replaces its test/QA/public_api_docs.jl (and its runtests.jl include of it) with nothing extra — the check now rides along with run_qa:

run_qa(MyPackage; explicit_imports = true)          # docstring check included by default
# add api_docs_kwargs = (; rendered = true) to also require rendering in docs/src

Only 2 repos actually grew the bespoke file (SciMLSensitivity.jl, DiffEqGPU.jl); their migration PRs are a follow-up blocked on a 2.0.0 release.

Validation

  • Pkg.test() green on Julia 1.11 (286 pass) and 1.10 (271 pass) locally — full output observed, not inferred.
  • Runic.jl formatting applied to the changed src/test files.
  • The suite includes a real, non-mocked assertion that SciMLTesting's own exported API is fully documented (run_api_docs(SciMLTesting) passes), so the check guards this package too.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AedZ1o7uENECZRF7xUwUaS

Add `run_api_docs(pkg)` and `public_api_names(pkg)` to SciMLTesting so the
public-API-docstring check that was being copy-pasted into individual SciML
repos as `test/QA/public_api_docs.jl` lives in one maintained place instead.

- `public_api_names(pkg)` — the sorted public API (exported names, plus
  `public` names on Julia >= 1.11), module name dropped.
- `run_api_docs(pkg; docstrings=true, rendered=false, ...)` — asserts every
  public name has a docstring (via `Base.Docs.doc` on the binding, so a
  re-export documented in its defining package counts), and optionally that
  each is rendered in a `@docs` block under `docs/src` (an `@autodocs` block
  satisfies it wholesale). `ignore`/`rendered_ignore` and
  `docstrings_broken`/`rendered_broken` handle exceptions and mid-migration
  repos.
- `run_qa` gains opt-in `api_docs` / `api_docs_kwargs`, so a repo's whole
  public-API-docs check collapses to `run_qa(MyPkg; api_docs = true)`.
- `REPL` is added as a (stdlib) dep and imported for its load-time side
  effect: on Julia >= 1.11 the `Base.Docs.doc(::Binding)` methods live in REPL
  and are absent in a bare `Pkg.test` process.

Tests cover `public_api_names`, the `@docs`/`@autodocs` scanner, the docstring
and rendered checks (pass/fail/ignore/broken paths via a ProbeTestSet), and the
`run_qa` integration — including a real assertion that SciMLTesting's own
exported API is fully documented. Verified locally on Julia 1.10 and 1.11.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Turn the public-API docstring check on by default in `run_qa` (`api_docs =
true`), so a plain `run_qa(MyPkg)` enforces that every exported/`public` name
has a docstring. Repos opt out per-call with `api_docs = false`, curate
exceptions via `api_docs_kwargs` (`ignore`, `docstrings_broken`), or document
their public API.

This changes the observable behavior of the existing `run_qa` for every caller
on upgrade (previously-green QA lanes with undocumented public API now fail), so
it is a breaking change: bump 1.9.0 -> 2.0.0. `explicit_imports` stays opt-in
(its per-repo ignore-lists make default-on impractical).

Docs (module docstring, run_qa/run_api_docs docstrings, README) updated to
describe the default-on behavior. Tests updated: the integration testset now
asserts the default runs the check (and that `api_docs = false` skips it), and
the JET-defaulting testset forces `api_docs = false` to stay focused. Verified
locally: Julia 1.11 (286 pass) and 1.10 (271 pass).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title Add run_api_docs: shared public-API documentation QA check Add run_api_docs and enforce public-API docstrings by default in run_qa (2.0.0) Jul 10, 2026
@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review July 10, 2026 01:00
@ChrisRackauckas
ChrisRackauckas merged commit 281ddef into main Jul 10, 2026
12 checks passed
@ChrisRackauckas
ChrisRackauckas deleted the api-docs-qa-check branch July 10, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants