Proposal for Backwards Compatibility & Releases

releases/ReleaseCompatibility.md

@@ -0,0 +1,157 @@
+# Releases & Compatibility
+
+To date, we have been somewhat cavalier about making breaking
+changes in Fairlearn.
+Although we are currently pre-`v1.0.0` and hence without particular
+commitments to compatibility (as generally understood - see e.g.
+the [Semantic Versioning Scheme](https://semver.org/)), we should
+work to reduce the number of breaking changes we make.
+The usual Python versioning scheme is
+[described in PEP440](https://www.python.org/dev/peps/pep-0440/).
+
+## The Problem
+
+At the time of writing, we have just released `v0.4.6`, which has
+some substantial breaking changes in the Metrics area of Fairlearn.
+Before it, `v0.4.5` reworked a smaller subset of this functionality, 
+caused smaller breakages.
+There are also further changes to Metrics planned, which may well
+cause further breaks (although these should be more minor).
+All this is obviously undesirable from a user standpoint - these look
+like 'patch' level releases, but are actually breaking their code.
+Concretely, we've had users install Fairlearn with `pip`, and then
+find themselves unable to run Notebooks from our GitHub project - not
+because the functionality was missing, but because it had been
+renamed.
+Moving our notebooks to being generated as part of the documentation
+[in our examples directory](https://github.com/fairlearn/fairlearn/tree/master/examples)
+will help with this, since this will result in the notebooks being
+versioned.
+However, improved backwards compatibility is still desirable.
+
+We do not need deprecation policies as elaborate of those of
+[SciKit-Learn](https://numpy.org/neps/nep-0023-backwards-compatibility.html)
+or [NumPy](https://numpy.org/neps/nep-0023-backwards-compatibility.html) - indeed,
+policies such as those would be excessive for Fairlearn, given the
+size of our code and user bases.
+However, we do need to start to move in that direction, or we will
+not be able to grow our user base due to chaos in the code.
+
+## Support Policy
+
+Starting with `v0.5.0` we should make a commitment that anything which works at `v0.n.m_0` will also work for `v0.n.m` so long as `m >= m_0`.
+However, we do *not* guarantee compatibility between `n` and `n+1` in this scheme (although we would seek to minimise breakage).
+If we have to do `v.n.m.post[i]` releases, we will only support the final `post` release in the chain.
+Note that according to
+[PEP440](https://www.python.org/dev/peps/pep-0440/#post-releases),
+`post[i]` releases should only incorporate documentation fixes, and
+**not** code fixes.
+
+In order to support less mature functionality, we should also add
+a `fairlearn.experimental` package (see [a similar namespace in
+SciKit-Learn](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.experimental)).
+Anything in there will be subject to breaking at any time.
+Using an `experimental` namespace would not have helped with our
+current set of breaks, since the changes were being made to core
+functionality.
+Rather, this is to give future developers a space where they can
+get feedback on new functionality without immediately being
+committed to supporting their speculative design decisions.
+
+At the time of writing, it appears that the dashboard code does
+not use namespaces.
+However, namespaces are supported in TypeScript, and as we develop
+the UX code, we should introduce a similar distinction.
+
+### Monitoring Backwards Compatibility
+
+To monitor the required backwards compatibility, each new release
+branch will have a pipeline for testing backwards compatibility
+associated with it.
+This pipeline will:
+1. Fetch Fairlearn itself from `master`
+1. Fetch the tests from the associated release branch
+1. Run the 'old' tests against the 'new' Fairlearn
+
+The question is then: which tests to run?
+One possible answer is the usual test suite run by the PR Gate
+(that is, the one under `tests/`).
+This could cause problems when there is a code fix which causes
+changes to 'golden values' in tests.
+While this should be a rare event (assuming we have set
+numerical tolerances appropriately), it could certainly happen
+and we would be left with the unpalatable prospect of having to
+fix the test in the release branch as well.
+
+As an alternative, we can run the contents of the 'old'
+`examples/` directory against the new Fairlearn package.
+If we are keeping to the backwards compatibility promise
+outlined above, then these 'old' examples should still run.
+The examples do not have any `assert` statements which could
+fail due to later bugfixes.
+The test coverage provided by the examples is not as
+comprehensive as that of the main test suite, but the examples
+do represent common user scenarios.
+
+The simplest way to run the examples would be to build the
+documentation (which runs them in order to capture text and
+graphical output).
+Alternatively, we could export the notebooks using the
+[utility in sphinx-gallery](https://sphinx-gallery.github.io/stable/utils.html#convert-python-scripts-into-jupyter-notebooks)
+and run the notebooks through `papermill`.
+This would allow us to add some small `asert` statements, in order
+to ensure that we don't have silent failures (as is done with
+the current notebook testing under `test/`).
+
+These new pipelines will become part of the PR Gate for `master`,
+except when we are moving from `v0.n` to `v0.n+1`.
+
+## Revised Branching and Release Policy
+
+Our recent releases have been made from `master`, without
+making a release branch.
+While this approach has desirable properties, using release branches
+will work better with the more robust support policy described above.
+
+For all the `master` and `release` branches, we will require a linear
+history in GitHub.
+This forbids plain `merge` commits, and we will prefer squash merges
+to rebases.
+
+We will keep the `master` branch at `v0.m.n.dev0` at all times,
+indicating that `master` is under active development (although we will
+always seek to keep `master` in a shippable condition).
+Releases will occur from branches.
+
+To create a new release:
+1. Create a branch `release/v.0.n.m`
+1. Remove the `dev0` suffix from the version on the release branch
+   - Bump `master` to `v0.n.m+1.dev0`
+1. Run the release pipeline on the new release branch
+1. Create a GitHub Release corresponding to the new package
+   (hopefully this can be automated)
+1. Create the new ADO pipeline to monitor backwards compatibility
+
+### Post Releases
+
+We should only put out `post[i]` releases for essential fixes (either
+to algorithms or code) - no new features are allowed.
+In general, fixes should be made in `master` and individually moved
+to the appropriate release branch as required.
+The move will probably best be done with `git rebase -i`
+(an interactive rebase), [as is the practice in
+`scikit-learn`](https://github.com/scikit-learn/scikit-learn/blob/master/doc/developers/maintainer.rst).
+This is in order to preserve a linear history on each branch.
+The exact procedure used for this will likely be updated by
+experience (specifically in the release instructions in the developer
+guide).
+After the release, create an appropriate GitHub release, and update
+the corresponding backwards compatibility pipeline.
+
+### Writing up Changes
+
+A related process alteration is needed to our `Changes.md` document.
+Currently, this is written both by and for the developers of Fairlearn.
+Going forward, we should ensure that it is readable by the *users* of Fairlearn.
+For the case of breaking changes, this means that any breaking
+change should be accompanied by migration instructions.