Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

CI workflows.

**Read this when:** a workflow that installs dependencies fails on one Node line but not another, when changing which npm writes `package-lock.json`, when changing how the Docker image is built, smoke-tested or published, or when `Unit tests: lmdb` hits its cap or a new unit suite seeds or clears rows.
**Read this when:** a workflow that installs dependencies fails on one Node line but not another, when changing which npm writes `package-lock.json`, when changing how the Docker image is built, smoke-tested or published, when `Unit tests: lmdb` hits its cap or a new unit suite seeds or clears rows, or when changing what the release cherry-pick decides to pick.

Index of every design note: [DESIGN.md](../DESIGN.md).

Expand All @@ -27,3 +27,9 @@ Pushes to `main` and release branches run the same staging path, copying into a
`test:unit:lmdb` re-runs `unitTests/resources`, `unitTests/apiTests` and `unitTests/bin` on LMDB, where every awaited commit waits on a disk sync. A fixture that seeds or clears one row per transaction costs seconds on the default engine and minutes on LMDB, and a runner with slow fsync multiplies exactly that part. `subscriptionPreviousCountScanBound.test.js` (harper#2826), `subscriptionSuperseded.test.js` (harper#2887) and `calibrationStore.test.js` each pushed the step past its cap this way. Nothing enforces this; the step's `timeout-minutes` is the only detector.

Write a fixture's rows in one transaction and await a write only before the step that reads it. Empty a table with `Table.clear()`, or with every delete in one transaction where the table has a full-text index (`clear()` refuses it) or the suite reads the deletes from the audit log (`clear()` writes none). Where a test checks write order (calibration discovery scans decisions by time), keep those writes serial and let the independent ones commit alongside them. Before raising the cap, compare each suite's time in the `lmdb` step with the same suite in `resources (core)`/`resources (indexes)` of the same job, and fix the suite whose LMDB time grew.

## A release cherry-pick skips a change its branch already has (`workflows/cherry-pick-patch.yml`, `scripts/change-landed.sh`)

Before picking, the cherry-pick job asks `scripts/change-landed.sh` whether picking the PR onto the release branch could land anything new. The script replays the picked commits onto the release in order with `git merge-tree`, resolving each conflict toward the commit, and the result must be the release's own tree. A conflict then means only that a commit met content a later commit of the PR already wrote. It also merges the PR's net change, `MERGE_BASE..HEAD_SHA` (`MERGE_SHA^1..MERGE_SHA` for a merge-commit merge), into the release, and that must leave the tree unchanged; this covers merge resolutions that no commit carries. Only that answer skips the PR, with no branch and no PR. A change whose file content is partly present, reverted, absorbed by main from another PR, or undone by a later commit of its own picks as before. So does a conflicting change, and so does a check that cannot run. "Unchanged" is decided by Git's merge rules, the same rules the picks themselves run under: a gitlink the release already moved past, or a configured merge driver that keeps the release's side, counts as present for both. Replaying a landed PR one commit at a time re-applies each intermediate commit against its own final content, which conflicts instead of coming out empty. harper#3067 was the second run of a `demilestoned`/`milestoned` pair re-picking harper#3038 onto the v5.3 the first run had just landed it on.

A hold is never cleared. When the check finds the change landed but a merge was excluded from the picks, the job reports it held, with no branch and no PR: a resolution that took one side outright leaves no diff in the net change, so "unchanged" cannot vouch for it. The fallback that builds from a lone merge commit is not provably the whole change, so it is never checked and keeps its hold. A PR whose merge of main rewrote its own commits' lines fails the replay merge, so after landing it is still re-picked into a conflict. So does a PR where one commit edits a file a later commit deletes and the release already has both: `-X theirs` only resolves content conflicts, not modify/delete, so the replay exits 1 at the edit and the job picks into a conflict PR whose net diff is empty. The check runs after the release branch is checked out, so it reads the same `.gitattributes` as the first pick; it gives up, and the job picks, when any picked commit changes a `.gitattributes` file, since later picks would read that copy. `unitTests/github/cherryPickPatch.test.mjs` runs the shipped job's steps against a fixture origin. The check needs git 2.45 or newer (`merge-tree` on trees); the runner images ship newer, and the suite fails under CI on older git rather than skipping.
46 changes: 46 additions & 0 deletions .github/scripts/change-landed.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Usage: change-landed.sh <target> <base> <head> <commit>...
# Exits 0 only when picking the commits onto <target> would land nothing new:
# replayed onto it with each conflict resolved toward the commit, they end at
# <target>'s own tree, and so does the net change <base>..<head>. 1 when they
# would land something, 2 when the check cannot tell; callers pick on either.
set -u

if [ "$#" -lt 4 ]; then
echo "::warning::change-landed.sh needs a target, base, head and at least one commit, so the commits will be picked"
exit 2
fi

TARGET="$1"
BASE="$2"
HEAD="$3"
shift 3

could_not_run() {
echo "::warning::could not check whether $TARGET already contains $BASE..$HEAD ($1), so it will be picked"
exit 2
}

TARGET_TREE=$(git rev-parse --verify --quiet "$TARGET^{tree}") || could_not_run "$TARGET does not resolve"

REPLAYED="$TARGET"
for COMMIT in "$@"; do
# A real pick reads the .gitattributes earlier picks wrote; merge-tree reads only the checkout's.
git diff-tree -z --name-only -r "$COMMIT^1" "$COMMIT" | grep -qzE '(^|/)\.gitattributes$' &&
could_not_run "$COMMIT changes .gitattributes"
REPLAYED=$(git merge-tree --write-tree -X theirs --merge-base="$COMMIT^1" "$REPLAYED" "$COMMIT")
case $? in
0) REPLAYED="${REPLAYED%%$'\n'*}" ;;
1) exit 1 ;;
*) could_not_run "git merge-tree failed replaying $COMMIT" ;;
esac
done
[ "$REPLAYED" = "$TARGET_TREE" ] || exit 1

MERGED=$(git merge-tree --write-tree --merge-base="$BASE" "$TARGET" "$HEAD")
case $? in
0) [ "${MERGED%%$'\n'*}" = "$TARGET_TREE" ] && exit 0 ;;
1) ;;
*) could_not_run "git merge-tree failed merging the net change" ;;
esac
exit 1
55 changes: 42 additions & 13 deletions .github/workflows/cherry-pick-patch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,7 @@ jobs:
run: |
cp .github/scripts/upsert-sticky-comment.js "$RUNNER_TEMP/upsert-sticky-comment.js"
cp .github/scripts/apply-picks.sh "$RUNNER_TEMP/apply-picks.sh"
cp .github/scripts/change-landed.sh "$RUNNER_TEMP/change-landed.sh"

- name: Configure git
run: |
Expand Down Expand Up @@ -311,9 +312,13 @@ jobs:
# The PR was merged with a merge commit: it IS the change.
PICK_FLAGS="-m 1"
PICK_SHAS="$MERGE_SHA"
CHANGE_BASE="${MERGE_SHA}^1"
CHANGE_HEAD="$MERGE_SHA"
else
PICK_FLAGS=""
PICK_SHAS=$(git rev-list --reverse --no-merges "${MERGE_BASE}..${HEAD_SHA}" | tr '\n' ' ')
CHANGE_BASE="$MERGE_BASE"
CHANGE_HEAD="$HEAD_SHA"
# Only merges a human DECIDED something in need holding, so a
# routine "merge main in, no conflicts" PR still lands
# automatically. The test is whether the recorded merge matches what
Expand All @@ -332,9 +337,11 @@ jobs:
if [ "$IS_MERGED" = "true" ]; then
# Nothing replayable: the PR's own content exists only in
# commits that cannot be replayed. The merge SHA gives a human
# something to review, but it is not provably the whole change.
# something to review, but it is not provably the whole change,
# so it cannot prove the whole change landed either.
PICK_SHAS="$MERGE_SHA"
HOLD="the PR has no replayable non-merge commits, so this branch was built from the merge commit alone"
CHANGE_HEAD=""
elif [ -n "$EXCLUDED_MERGES" ]; then
echo "pushed=false" >> "$GITHUB_OUTPUT"
# Nothing replayable and a merge that resolved something: the
Expand All @@ -350,26 +357,48 @@ jobs:
fi
fi
fi
echo "pick_flags=$PICK_FLAGS" >> "$GITHUB_OUTPUT"
echo "pick_shas=$PICK_SHAS" >> "$GITHUB_OUTPUT"

if [ -n "$EXCLUDED_MERGES" ]; then
HOLD=$(printf 'merge commit(s) `%s` resolved content of their own and cannot be replayed' "$EXCLUDED_MERGES")
fi
echo "excluded_merges=$EXCLUDED_MERGES" >> "$GITHUB_OUTPUT"
echo "hold=$HOLD" >> "$GITHUB_OUTPUT"

# ── Create / reset cherry-pick branch off release branch ────────
git checkout -B "$BRANCH" "origin/$RELEASE_BRANCH"

# ── Skip a change the release branch already has ────────────────
#
# Replaying a landed change commit by commit conflicts: each
# intermediate commit meets its own final content. A hold still
# stands: a resolution that took one side outright leaves nothing in
# the net change for the check to find.
# shellcheck disable=SC2086
if [ -n "$CHANGE_HEAD" ] && "$RUNNER_TEMP/change-landed.sh" "origin/$RELEASE_BRANCH" "$CHANGE_BASE" "$CHANGE_HEAD" $PICK_SHAS; then
echo "The change from PR #$PR_NUMBER is already on $RELEASE_BRANCH — nothing to pick"
if [ -n "$HOLD" ]; then
echo "hold=its change appears to be on it already, but $HOLD" >> "$GITHUB_OUTPUT"
fi
echo "no_op=true" >> "$GITHUB_OUTPUT"
exit 0
fi

echo "pick_flags=$PICK_FLAGS" >> "$GITHUB_OUTPUT"
echo "pick_shas=$PICK_SHAS" >> "$GITHUB_OUTPUT"
echo "excluded_merges=$EXCLUDED_MERGES" >> "$GITHUB_OUTPUT"
echo "hold=$HOLD" >> "$GITHUB_OUTPUT"

# ── Apply commits, tracking which conflicted ────────────────────
CONFLICTS=$("$RUNNER_TEMP/apply-picks.sh" "$PICK_FLAGS" "$PICK_SHAS")
echo "conflicts=$CONFLICTS" >> "$GITHUB_OUTPUT"

if [ -z "$CONFLICTS" ] && [ -z "$HOLD" ] && [ -z "$(git rev-list "origin/${RELEASE_BRANCH}..HEAD")" ]; then
echo "Nothing to land on $RELEASE_BRANCH — every commit was already present"
echo "no_op=true" >> "$GITHUB_OUTPUT"
# An earlier run may have pushed a branch that is now redundant.
if [ -z "$CONFLICTS" ] && [ -z "$(git rev-list "origin/${RELEASE_BRANCH}..HEAD")" ]; then
# Nothing landed on this branch, so there is nothing to push a
# human could review either — a hold does not change that. An
# earlier run may have pushed a branch that is now redundant.
git push origin --delete "$BRANCH" 2>/dev/null || true
if [ -z "$HOLD" ]; then
echo "Nothing to land on $RELEASE_BRANCH — every commit was already present"
echo "no_op=true" >> "$GITHUB_OUTPUT"
fi
exit 0
fi

Expand All @@ -392,10 +421,10 @@ jobs:
--base "$RELEASE_BRANCH" --head "$BRANCH" --json number --jq '.[0].number' 2>/dev/null || true)
if [ -n "$STALE_PR" ] && [ "$STALE_PR" != "null" ]; then
gh pr close "$STALE_PR" --repo "$GITHUB_REPOSITORY" \
--comment "Superseded: PR #${PR_NUMBER} no longer produces any commits to cherry-pick onto \`${RELEASE_BRANCH}\`." >/dev/null || true
--comment "Superseded: PR #${PR_NUMBER} has nothing left to cherry-pick onto \`${RELEASE_BRANCH}\`." >/dev/null || true
fi
git push origin --delete "$BRANCH" 2>/dev/null || true
BODY=$(printf '%s\n## Release cherry-pick `%s`: already present\n\nEvery commit from this PR is already on `%s` — nothing to cherry-pick.\n' \
BODY=$(printf '%s\n## Release cherry-pick `%s`: already present\n\nThe change from this PR is already on `%s` — nothing to cherry-pick.\n' \
"$STICKY_MARKER" "$RELEASE_BRANCH" "$RELEASE_BRANCH")
node "$RUNNER_TEMP/upsert-sticky-comment.js" "$PR_NUMBER" "$STICKY_MARKER" "$BODY"

Expand Down Expand Up @@ -487,10 +516,10 @@ jobs:
--base "$RELEASE_BRANCH" --head "$BRANCH" --json number --jq '.[0].number' 2>/dev/null || true)
if [ -n "$STALE_PR" ] && [ "$STALE_PR" != "null" ]; then
gh pr close "$STALE_PR" --repo "$GITHUB_REPOSITORY" \
--comment "Superseded: PR #${PR_NUMBER} no longer produces any commits to cherry-pick onto \`${RELEASE_BRANCH}\`." >/dev/null || true
--comment "Superseded: PR #${PR_NUMBER} has nothing left to cherry-pick onto \`${RELEASE_BRANCH}\`." >/dev/null || true
fi
git push origin --delete "$BRANCH" 2>/dev/null || true
BODY=$(printf '%s\n## Release cherry-pick `%s`: held for review\n\nNothing could be cherry-picked onto `%s`: %s.\n\n`%s` was **not** updated. Apply this change by hand if it belongs on that line.\n' \
BODY=$(printf '%s\n## Release cherry-pick `%s`: held for review\n\nNothing could be cherry-picked onto `%s`: %s.\n\n`%s` was **not** updated. Check it, and apply this change by hand if any of it is missing and belongs on that line.\n' \
"$STICKY_MARKER" "$RELEASE_BRANCH" "$RELEASE_BRANCH" "$HOLD" "$RELEASE_BRANCH")
node "$RUNNER_TEMP/upsert-sticky-comment.js" "$PR_NUMBER" "$STICKY_MARKER" "$BODY"
exit 0
Expand Down
1 change: 1 addition & 0 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Index of the design notes for the harper core: one line per note, grouped by the
- [The lock file must pass `npm ci` under npm 10 and npm 11 (`workflows/lockfile-npm-compat.yml`)](.github/DESIGN.md#the-lock-file-must-pass-npm-ci-under-npm-10-and-npm-11-workflowslockfile-npm-compatyml) — npm 11 writes the lock but accepts some that npm 10's `npm ci` rejects; fix the conflict at its source, never by regenerating with npm 10.
- [A release copies to Docker Hub only the image it booted (`workflows/docker-smoke.yml`, `workflows/publish-docker.yaml`)](.github/DESIGN.md#a-release-copies-to-docker-hub-only-the-image-it-booted-workflowsdocker-smokeyml-workflowspublish-dockeryaml) — The release builds once into a runner-local registry, smoke-tests that digest and copies the same index to Docker Hub; never a rebuild.
- [Unit-test fixtures don't spend `Unit tests: lmdb` on one durable commit per row (`workflows/unit-test.yml`)](.github/DESIGN.md#unit-test-fixtures-dont-spend-unit-tests-lmdb-on-one-durable-commit-per-row-workflowsunit-testyml) — Every resources suite re-runs on LMDB, where each awaited commit syncs; fixtures batch their writes, and the step cap is the only detector.
- [A release cherry-pick skips a change its branch already has (`workflows/cherry-pick-patch.yml`, `scripts/change-landed.sh`)](.github/DESIGN.md#a-release-cherry-pick-skips-a-change-its-branch-already-has-workflowscherry-pick-patchyml-scriptschange-landedsh) — skip only when replaying the PR's commits onto the release, conflicts resolved toward each commit, ends at the release's own tree; replaying a landed PR commit by commit conflicts instead.

## resources/ — records, transactions, tables, blobs, audit

Expand Down
Loading
Loading