Add run_api_docs and enforce public-API docstrings by default in run_qa (2.0.0) - #19
Merged
Conversation
Add `run_api_docs(pkg)` and `public_api_names(pkg)` to SciMLTesting so the public-API-docstring check that was being copy-pasted into individual SciML repos as `test/QA/public_api_docs.jl` lives in one maintained place instead. - `public_api_names(pkg)` — the sorted public API (exported names, plus `public` names on Julia >= 1.11), module name dropped. - `run_api_docs(pkg; docstrings=true, rendered=false, ...)` — asserts every public name has a docstring (via `Base.Docs.doc` on the binding, so a re-export documented in its defining package counts), and optionally that each is rendered in a `@docs` block under `docs/src` (an `@autodocs` block satisfies it wholesale). `ignore`/`rendered_ignore` and `docstrings_broken`/`rendered_broken` handle exceptions and mid-migration repos. - `run_qa` gains opt-in `api_docs` / `api_docs_kwargs`, so a repo's whole public-API-docs check collapses to `run_qa(MyPkg; api_docs = true)`. - `REPL` is added as a (stdlib) dep and imported for its load-time side effect: on Julia >= 1.11 the `Base.Docs.doc(::Binding)` methods live in REPL and are absent in a bare `Pkg.test` process. Tests cover `public_api_names`, the `@docs`/`@autodocs` scanner, the docstring and rendered checks (pass/fail/ignore/broken paths via a ProbeTestSet), and the `run_qa` integration — including a real assertion that SciMLTesting's own exported API is fully documented. Verified locally on Julia 1.10 and 1.11. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Turn the public-API docstring check on by default in `run_qa` (`api_docs = true`), so a plain `run_qa(MyPkg)` enforces that every exported/`public` name has a docstring. Repos opt out per-call with `api_docs = false`, curate exceptions via `api_docs_kwargs` (`ignore`, `docstrings_broken`), or document their public API. This changes the observable behavior of the existing `run_qa` for every caller on upgrade (previously-green QA lanes with undocumented public API now fail), so it is a breaking change: bump 1.9.0 -> 2.0.0. `explicit_imports` stays opt-in (its per-repo ignore-lists make default-on impractical). Docs (module docstring, run_qa/run_api_docs docstrings, README) updated to describe the default-on behavior. Tests updated: the integration testset now asserts the default runs the check (and that `api_docs = false` skips it), and the JET-defaulting testset forces `api_docs = false` to stay focused. Verified locally: Julia 1.11 (286 pass) and 1.10 (271 pass). Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
ChrisRackauckas
marked this pull request as ready for review
July 10, 2026 01:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This draft PR should be ignored until reviewed by @ChrisRackauckas.
Motivation
A wave of near-identical PRs (SciML/SciMLSensitivity.jl#1537, SciML/DiffEqGPU.jl#472) added a hand-copied
test/QA/public_api_docs.jlto individual SciML repos, each re-implementing the same check: every exported name has a docstring (and is rendered in the docs). This PR factors that into the shared harness and turns it on by default, so the fleet enforces public-API docstrings from a plainrun_qa(MyPkg)and repos delete their bespoke files.What's added
public_api_names(pkg)— the sorted public API: exported names, pluspublic-declared names on Julia >= 1.11 (exported-only on the 1.10 LTS, so no per-repoif VERSIONguards).run_api_docs(pkg; docstrings=true, rendered=false, docs_src, ignore, rendered_ignore, docstrings_broken, rendered_broken):Base.Docs.docon the binding, so a name re-exported from a dependency counts as documented via that dependency — a repo isn't forced to redocument re-exports.```@docsblock underdocs_src(defaults to<pkgroot>/docs/src). An```@autodocsblock satisfies it wholesale.ignore/rendered_ignorefor exceptions;docstrings_broken/rendered_brokento mark the check@test_brokenfor a mid-migration repo (auto-flags an Unexpected Pass once fully documented).run_qanow runsapi_docsby default (api_docs = true) viaapi_docs_kwargs.explicit_importsstays opt-in (its per-repo ignore-lists make default-on impractical).REPLadded as a (stdlib) dep, imported for its load-time side effect: on Julia >= 1.11 theBase.Docs.doc(::Binding)methods live in REPL and are absent in a barePkg.testprocess. (The copied per-repo files only worked because the test env happened to load REPL transitively.)Breaking change -> 2.0.0
Turning
api_docson by default changes the observable behavior of the pre-existingrun_qafor every caller: a repo that upgrades and has undocumented public API will see its QA lane go red where it was green. Per SemVer that is a breaking change, so this is a major bump to 2.0.0 (not 1.9.0). Escape hatches:run_qa(MyPkg; api_docs = false), orapi_docs_kwargs = (; ignore = (...))/(; docstrings_broken = true).Migration path for repos
A repo replaces its
test/QA/public_api_docs.jl(and itsruntests.jlinclude of it) with nothing extra — the check now rides along withrun_qa:Only 2 repos actually grew the bespoke file (SciMLSensitivity.jl, DiffEqGPU.jl); their migration PRs are a follow-up blocked on a 2.0.0 release.
Validation
Pkg.test()green on Julia 1.11 (286 pass) and 1.10 (271 pass) locally — full output observed, not inferred.src/testfiles.run_api_docs(SciMLTesting)passes), so the check guards this package too.🤖 Generated with Claude Code
https://claude.ai/code/session_01AedZ1o7uENECZRF7xUwUaS