Fix ccache usage in GitHub Actions

[jschmidt-icinga]

This fixes the actions/cache GitHub action we use to save and restore ccache for faster builds in PRs that change little or no code.

As a rough estimate, the Linux workflow on this PR, when rebuilt with a warm cache went down from 2h 17m to 1h 25m.

What was broken?

The default cache action only saves to a key when it didn't find an exact match. On current master the save and restore keys are identical and depend on just the matrix.distro string. This effectively means one cache entry per distro got written (probably a long time ago), that old entry always gets restored again and again, only ever leads to cache misses and no new entry gets saved because the keys matched exactly. This is the reason we weren't seeing any benefits, even when force-pushing to a PR with zero code changes.

To solve this we now save to a key in the format ccache-<distro>-<platform>-<runid> with the runid being the critical part, and we use the restore-keys attribute of the action to do a prefix-matched restore from any key that matches just the distro and platform. This means that a new cache entry gets stored for each run.

Why save only on pushes to master

GitHub actions/cache has a total limit of 10 GiB across all cache entries per repo. The eviction for old entries past this limit is LRU. Meanwhile ccache's limits are controlled via the CCACHE_MAXSIZE environment variable or otherwise defaults to 5 GiB. Lets assume for a moment we left it at the default and each distro accumulated more and more cached objects. At some point, the total objects cached between alpine:bash and debian:12 (linux/386) (the first and last jobs at the time of writing) will weigh enough that they will go over the GHA's maximum of 10 GiB and alpine:bash's cache of the same run will already be evicted by GitHub causing full rebuilds on their next run. Then amazonlinux:2023 gets evicted and so on.

So we set CCACHE_MAXSIZE to a reasonable limit for all distros. I've chosen 400 MiB, which fits around four full rebuilds for the average distro (each weighing around 100 MiB), which currently gives us 8 GiB if all distros fully use it, so a bit of headroom remains for other things or additions to the matrix. This avoids any individual run evicting caches of earlier jobs. But there is still a problem when CI jobs for multiple PRs run concurrently, which is very common in this repo. There is no sane way to cap this either, without restricting the max size further into uselessness.

So the idea is to always restore the cache from the last master build and save it only on a push to master after a PR has been merged. Since those don't run concurrently, we ensure that there is always a cache that can be read for PR jobs. Obviously how useful that cache is will depend on how much that PR changes (more on that further down).

Invoking ccache through CMake

Previously we were using (or trying to anyway) ccache's compiler wrapper scripts by prioritizing them in $PATH. This has some issues with compatibility that we can afford to avoid because CMake has native support for launchers like ccache since version 3.17. By setting CMAKE_<lang>_COMPILER_LAUNCHER=ccache it just works. The build scripts will now just preface every compiler invocation with the ccache command.

Other changes

• Amazonlinux doesn't have ccache in its regular repos, but it does have it in its SPAL repos, which can be installed through the spal-release package, so we do that now.
• I've disabled the debug symbols that get added through some of the distros' flags by appending -g0 to the very end of the compiler flags. This about halves the cache space required by each run. I don't think debug symbols add anything in the GitHub CI, so that's a good trade-off in my eyes.

Future improvements

My idea for the future, once #10936 gets merged is to look into enabling precompiled headers in a separate PR. That would allow us reasonably fast non-unity builds, which would synergize much better with the cache action, because it would be more efficient with the cache size and rebuilds would only ever be what the PR directly touches, making it less of a hassle that we only ever restore the cache from master.

[Al2Klimov]

Purrs like a cat:

https://github.com/Al2Klimov/icinga2/actions/runs/35071225210/job/104712961499?pr=17

[julianbrost]

Purrs like a cat:

https://github.com/Al2Klimov/icinga2/actions/runs/35071225210/job/104712961499?pr=17

Would you mind sharing a bit more on what you tested there? To me, it's not really obvious what to look for there.

[Al2Klimov]

Well, that PR-run behaves exactly as promised: a ccache gets restored. While on it, its build takes a few minutes less than the master one.

[backbot-ci]

Backport failed for support/2.15, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin support/2.15
git worktree add -d .worktree/backport-10966-to-support/2.15 origin/support/2.15
cd .worktree/backport-10966-to-support/2.15
git switch --create backport-10966-to-support/2.15
git cherry-pick -x 53dcabd2c853bb29e8243a0748f178dfa18b1a44 e64ef211cfb118378f4d1e9669a1d6a5eec07851 11dbd2bfedfd74340cabf1300964386f69550362

[backbot-ci]

Backport failed for support/2.15, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin support/2.15
git worktree add -d .worktree/backport-10966-to-support/2.15 origin/support/2.15
cd .worktree/backport-10966-to-support/2.15
git switch --create backport-10966-to-support/2.15
git cherry-pick -x 53dcabd2c853bb29e8243a0748f178dfa18b1a44 e64ef211cfb118378f4d1e9669a1d6a5eec07851 11dbd2bfedfd74340cabf1300964386f69550362

[backbot-ci]

Successfully created backport PR for support/2.16:

• [Backport support/2.16] Fix ccache usage in GitHub Actions #11037

[Al2Klimov]

v2.15 misses c34e030, hence the conflict.