proposal: Pin to specific PHPUnit minor versions

[acoulton]

This is for discussion, as there are pros and cons.

As with the original Behat implementation, all the code in this extension is coupled to internal PHPUnit classes and methods (and, if we add support for 11 - 13, internal state). We also now have confirmation from Sebastian Bergmann that using PHPUnit assertions outside the PHPUnit runtime is explicitly not supported.

So, just because our CI proves that the extension supports PHPUnit 13.3.4, there is no guarantee that it will support 13.3.5 or any future minor or bugfix PHPUnit release.

One option is:

• Our composer.json explicitly supports one exact version per PHP major (as in this PR).
• We build a script and CI job similar to the Behat/Gherkin update job. This would check for new PHPUnit releases, bump our composer.json pins, and send us a PR.
• If / when the PR is green, we release a new version of our library.
• End-users would then upgrade this library and their PHPUnit version in sync with each other.

Advantages

The biggest advantage of this is guaranteed build stability. End-users will only ever be able to install a pairing of this extension & a PHPUnit version that are known to work together. If a new PHPUnit minor introduces a new incompatibility, nobody will get that in CI until we've updated this library to support it.

Another advantage is that minimising the number of PHPUnit versions we support in parallel will keep the library simpler, and reduce the amount of hacking around different versions of PHPUnit internals that we have to do.

Disadvantages

The biggest disadvantage is that it puts us in the critical path if PHPUnit release a security fix. Nobody using this library would be able to update to the new PHPUnit until we merge and release - even if the new version works "out of the box" for us.

In theory, people could alias the PHPUnit in their composer.json (e.g. 13.3.5 as 13.3.4) but that is very hacky and wouldn't be suggested by e.g. Dependabot or other automated flows so would still slow down the upgrade.

Another disadvantage is the maintainer load for us - even if we automate creating the PRs, and creating the releases, we would still need a human to merge / approve at least once per release.

Alternative - pin to a minor

One compromise is to pin to the minor version (e.g. 8.5 || 9.6 || 10.5 || 11.5 || 12.5 || 13.3). That would be better than nothing for build stability, would be fewer releases than a full pin, and reduces the risk of holding up a security release. But it's not perfect - there could still be changes that are breaking for us in a bugfix, and there could be a security release that goes in a new minor.

Alternative - recommend that end-users pin, and 🤞

The simplest solution would be to strongly advise end users to pin their own projects to a specific version of PHPUnit and this library (in their composer.lock, or in composer.json for libraries that don't commit the lockfile).

That should provide a reasonable level of build stability, while still letting them be in control of upgrading PHPUnit. But:

• It doesn't eliminate the risk of an unexpected CI failure - it's possible that a build is green if all their steps pass on a version update commit, but unexpectedly fails later when an assertion fails.
• It doesn't eliminate the possibility that updating PHPUnit causes their build to fail, and that they can't fix that without reworking their tests or getting a new version of this library. So we could still be blocking a security update, just less visibly...

[acoulton]

I think I'm leaning to the last option - NOT pinning anything in this library, and just making it very clear that end-user projects should definitely pin their dependencies so that they can control the timing of PHPUnit version bumps.

But I'm keen to hear other opinions.

Worth saying that if we start off pinning, we could relax that later.

But if we do a first release with e.g. ^13.3, that's pretty much locked in. If we later tried to tighten that to a more specific version / range, there's a high chance composer would install the newer PHPUnit and roll back to the older, less restrictive, version of our library.

[carlos-granados]

I think I'm leaning to the last option - NOT pinning anything in this library, and just making it very clear that end-user projects should definitely pin their dependencies so that they can control the timing of PHPUnit version bumps.

But I'm keen to hear other opinions.

Worth saying that if we start off pinning, we could relax that later.

But if we do a first release with e.g. ^13.3, that's pretty much locked in. If we later tried to tighten that to a more specific version / range, there's a high chance composer would install the newer PHPUnit and roll back to the older, less restrictive, version of our library.

I'm in favour of not pinning anything explicitly. Apart from all the reasons you cited there's another thing to take into account. Suppose we pin to 13.5. Then another dependency says that they need 13.6. Then you can't update this second dependency because we're blocking it. If had this in the past with other projects and it's quite frustrating. So my vote goes for the not pinning option

[acoulton]

Thanks @carlos-granados the more I think about it the more I agree.

I'd envisaged with automated bumping of the pin we'd only be a few days behind a PHPUnit release at most, so the conflict you describe would be short-lived. But it could still be frustrating, and it's another pressure point on us to review & release updates quickly.

I think ultimately we just have to accept (and make clear) that we're using PHPUnit in an unsupported way - and there's no magic thing we can do that will eliminate the risk of a PHPUnit update breaking your CI... Which is why we only recommend this for projects with existing suites that they can't migrate to some other assertion tool.