activate_group_env mutates the repo's tracked test/qa/Project.toml on every QA run

[ChrisRackauckas-Claude]

Summary

activate_group_env activates a git-tracked, in-repo directory as the active project and then calls Pkg.develop into it. Pkg.develop writes to the activated Project.toml, so every QA run mutates a tracked file in the repo under test. The tree is left dirty, and it is easy to commit the churn by accident along with unrelated work.

Reproduction

Fresh shallow clone of SciML/OrdinaryDiffEq.jl at 7b1ad03, clean tree (git status --porcelain empty), Julia 1.12.6, SciMLTesting v2.6.2:

$ cd lib/OrdinaryDiffEqBDF
$ GROUP=QA julia --project=. -e 'using Pkg; Pkg.test()'
$ cd ../..
$ git status --porcelain
 M lib/OrdinaryDiffEqBDF/test/qa/Project.toml

--- a/lib/OrdinaryDiffEqBDF/test/qa/Project.toml
+++ b/lib/OrdinaryDiffEqBDF/test/qa/Project.toml
@@ -2,21 +2,22 @@
 AllocCheck = "9b6a8646-10ed-4001-bbdc-1d2f46dfbb1a"
 Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
 JET = "c3a54625-cd67-489e-a8e7-0a5a0ff4e31b"
+OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
 OrdinaryDiffEqBDF = "6ad6398a-0878-4a85-9266-38940aa047c8"
 OrdinaryDiffEqCore = "bbf590c4-e513-4bbe-9b18-05decba2e5d8"
 SciMLBase = "0bca4576-84f4-4d90-8ffe-ffa030f20462"
 SciMLTesting = "09d9d899-5365-40a9-917a-5f67fddea283"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
-[sources.OrdinaryDiffEqCore]
-path = "../../../OrdinaryDiffEqCore"
+[sources]
+OrdinaryDiffEqCore = {path = "../../../OrdinaryDiffEqCore"}
 
 [compat]
 AllocCheck = "0.2"
 Aqua = "0.8.11"
 JET = "0.9, 0.11"
-SciMLTesting = "2.1"
-julia = "1.10"
 OrdinaryDiffEqBDF = "2"
 OrdinaryDiffEqCore = "4"
 SciMLBase = "3"
+SciMLTesting = "2.1"
+julia = "1.10"

Three distinct edits: a new [deps] entry, a rewritten [sources] table style, and reordered [compat] keys.

Mechanism

activate_group_env (src/SciMLTesting.jl):

Pkg.activate(group_dir)
...
specs = [Pkg.PackageSpec(path = abspath(p)) for p in parents]
isempty(specs) || Pkg.develop(specs)
develop_sources && develop_sources!(group_dir; parent = parent)
instantiate && Pkg.instantiate()

group_dir here is lib/<sublib>/test/qa, which is tracked. Pkg.develop adds each parent to [deps] and rewrites the file, and Pkg normalizes the whole TOML while it is at it.

Note the OrdinaryDiffEq entry comes from the caller passing two parents — OrdinaryDiffEq's test/runtests.jl does:

activate_group_env(joinpath(@__DIR__, "qa");
    parent = [dirname(@__DIR__), joinpath(@__DIR__, "..", "..", "..")])

i.e. the sublibrary and the repo root. With the default single parent the file happens not to change, because the sublibrary is already listed — so this does not reproduce via activate_group_env with default arguments; you need the real call, or any case where a parent is not already in [deps].

Why it matters beyond tidiness

The injected OrdinaryDiffEq lands in [deps] with no corresponding [sources] entry (the dev path is recorded in the gitignored Manifest.toml). If that diff is ever committed, the QA env declares a dependency on the registered umbrella package rather than the in-repo one — a monorepo QA environment silently testing against a released version of the thing it lives inside. That is a much worse outcome than the noise itself.

It also means a contributor running QA locally cannot distinguish their own changes from harness churn in git status, and git stash-based "does this fail on master too?" checks pick up the mutation.

Possible fixes

1. Do not mutate the repo. Copy the group env to a temporary directory, activate that, and develop there. Cleanest, and makes QA runs read-only with respect to the working tree.
2. Restore afterwards. Snapshot Project.toml before the Pkg.develop/instantiate and write it back in a finally. Simple, but leaves a window and loses the value of the resolved state.
3. Do not develop what is already resolvable. Only Pkg.develop parents that are not already in [deps] with a usable [sources] path — narrows the blast radius but does not eliminate it, and does not stop Pkg's TOML normalization.

(1) seems right. Whichever is chosen, it is worth checking whether activate_gpu_env-style helpers and the non-QA group envs have the same problem.

Environment

Julia 1.12.6
SciMLTesting v2.6.2
OrdinaryDiffEq.jl at 7b1ad03

Found while verifying unrelated QA failures. Unrelated to, but easily confused with, the public API has docstrings failure fixed in #45.