Add --extra-repo-priority for APT pinning + regression test

[Simon Beaudoin (simonbeaudoin0935)]

Adds --extra-repo-priority, paired positionally with --extra-repo, to let a repo win APT dependency resolution even when a different source offers a higher version number. Implemented by writing an APT preferences file into the sbuild chroot (pinned by origin/hostname) via --chroot-setup-commands, since sbuild has no built-in pinning option.

Also adds a real regression test (test/pinning/) reproducing the actual bug: two throwaway containers each serve a conflicting version of a synthetic libqcomdummy-dev package, and a fixture source package fails to build unless the intended version resolves. The test runs the build twice - once unpinned (expected to fail, proving the bug) and once with --extra-repo-priority (expected to succeed, proving the fix) - wired into container-build-and-upload.yml as a job that runs after build-deb-arm64.

Also moves the existing dummy-package CI fixture from .github/dummy-package/ to test/dummy-package/, keeping .github/ limited to GitHub-special content, alongside the new test/pinning/ fixture.

Related Issue
Fixes #48

[bkosuri]

Verified build of gst-plugins-good1.0 which depends on gst-plugins-base1.0.
Upstream version of gst-plugins-base1.0: 1.26.2-1+deb13u2
Version in CLO artifactory: 1.26.2-1+deb13u1+qcom1

When --extra-repository-priority is set, CLO version got picked up.

[Loïc Minier (lool)]

I see the PR has:

• main implementation commit => I'll review the approach here
• docs/tests updates => skipping this for now
• bug fixes to earlier commits => please squash into the earlier commits

The main implementation currently generates an APT preference file with a particular priority during the docker pkg build, based on the hostname passed as an URL in an earlier extra-repo flag. I believe this is to support builds where CLO archive is in sources and is preferred over Debian. The flags of docker_deb_build are based on sbuild's flags, but since sbuild doesn't provide such an APT preference flag, a --chroot-setup-commands flag is used instead.

The high-level strategy is ok, things I am not comfortable with:

1. what happens if someone passes multiple extra-repository flags with different hostnames (I don't even know if sbuild supports that)

2. how do we ensure people using this get the same behavior, e.g. how do we convey the set of flags to use for this or that typical build environment? It's fine to provide a low-level tool for people to do what they want, we also want a high-level tool that will ensure that different people build the packages in the same way. Perhaps this is more for qcom-build-utils / qli-ci, I'm not sure, but is there a place where we would define build environments profiles? Perhaps something like:

• ubuntu: build against latest Ubuntu stronking (or other suites, or Debian etc.)
• qli: build against Debian trixie + QLI APT repo
• clo: build against Debian trixie + QLI APT repo + CLO APT repo

3. hostnames will work to pin the repo, there are often other interesting properties in the Release file to use for APT preferences, so:
   a) are there other properties that could be used for the CLO repo?
   b) how do we pass other APT preferences data or other configs? would it make more sense to simply channel sbuild's flags such as --chroot-setup-commands in the low-level wrapper?

[Simon Beaudoin (simonbeaudoin0935)]

Thanks for the thorough review!

1. Multiple --extra-repo with different hostnames does work today - test/pinning/run-pinning-test.sh exercises exactly that (two repos, two hostnames, one pinned) and it's green in CI. -e/--extra-repo was already action='append' before this PR, and build_apt_pin_commands has no hardcoded arity, so it should scale to N repos fine.

2. Agreed on build profiles being useful, but I'd lean towards keeping that out of this PR - docker-pkg-build is meant to stay a low-level wrapper, and named profiles (ubuntu/qli/clo -> the right flags) feel more at home in qli-ci or qcom-build-utils. Happy to file a follow-up issue there if you agree.

3. Good push on this - I checked the Release files this repo already points at: the QArtifactory-backed sources (qsc-deb-releases) publish Origin: Artifactory / Label: Artifactory, which is just JFrog's generic default, not Qualcomm-specific. The Debusine qli repo does have a real Origin: Qualcomm / Label: qli though. So whether release-based pinning is even a good idea depends on which backend actually serves the CLO archive - if it's the same Artifactory instance, pinning on origin=Artifactory would be too broad (matches every repo on that instance), and hostname pinning is actually the safer bet. Do you know which one hosts CLO? That'll tell us if it's worth supporting release-attribute pinning too.

On the passthrough idea (3b): open to adding a raw --chroot-setup-commands passthrough alongside --extra-repo-priority, for cases where hostname pinning isn't the right tool. Wanted to check with you first before adding more surface area, since it'd need its own thought about safe quoting (this PR's pin content is base64-wrapped specifically to survive the existing nested-quoting between bash -c and gbp --git-builder="...", which a raw passthrough wouldn't get for free).

[Loïc Minier (lool)]

Thanks for the thorough review!

1. Multiple --extra-repo with different hostnames does work today - test/pinning/run-pinning-test.sh exercises exactly that (two repos, two hostnames, one pinned) and it's green in CI. -e/--extra-repo was already action='append' before this PR, and build_apt_pin_commands has no hardcoded arity, so it should scale to N repos fine.

So now the implementation will setup an APT pin for each extra-repo, but what if I specify two times the same host in two separate extra-repo (e.g. APT repo qli + qli-staging on artifactory) but with two priorities? which APT preferences will win, since the only criteria is the hostname?

Also, this relies on documentation + user behaving properly, there's no enforcement to have as many --extra-repo as --extra-repo-priority.

2. Agreed on build profiles being useful, but I'd lean towards keeping that out of this PR - docker-pkg-build is meant to stay a low-level wrapper, and named profiles (ubuntu/qli/clo -> the right flags) feel more at home in qli-ci or qcom-build-utils. Happy to file a follow-up issue there if you agree.

Sure, as long as there's a clear way to implement this for tech teams that need the CI setup right.

3. Good push on this - I checked the Release files this repo already points at: the QArtifactory-backed sources (qsc-deb-releases) publish Origin: Artifactory / Label: Artifactory, which is just JFrog's generic default, not Qualcomm-specific. The Debusine qli repo does have a real Origin: Qualcomm / Label: qli though. So whether release-based pinning is even a good idea depends on which backend actually serves the CLO archive - if it's the same Artifactory instance, pinning on origin=Artifactory would be too broad (matches every repo on that instance), and hostname pinning is actually the safer bet. Do you know which one hosts CLO? That'll tell us if it's worth supporting release-attribute pinning too.

You can also pin on suite, which I hope Artifactory would set? CLO is on a separate artifactory instance, I believe at https://artifacts.codelinaro.org/

On the passthrough idea (3b): open to adding a raw --chroot-setup-commands passthrough alongside --extra-repo-priority, for cases where hostname pinning isn't the right tool. Wanted to check with you first before adding more surface area, since it'd need its own thought about safe quoting (this PR's pin content is base64-wrapped specifically to survive the existing nested-quoting between bash -c and gbp --git-builder="...", which a raw passthrough wouldn't get for free).

So I'm looking at this from a distance. IIUC, this is the low-level part of the package builds for Ubuntu CI and devcompute use cases. For end-developers, we want this to be flexible (hence the idea of expanding here), and there I think we want to channel the flexibility of sbuild, with perhaps some sensible defaults for us. Kind of a qcom-docker-sbuild-for-pkg-builds. For CI, we want rigorous definition for reproducibility.

Since this PR is patching the low-level tool and not about the high-level, and since the wrapper already channels a number of sbuild options, perhaps you want to channel some more, or all of sbuild?

BTW sbuild already does "build in containers" in some way, perhaps we could even get rid of the docker side of things, not sure of the use cases we're trying to cover.

[Simon Beaudoin (simonbeaudoin0935)]

Good pushback, and it made me actually test something instead of assuming.

1. Small correction first: count mismatch between --extra-repo and --extra-repo-priority is enforced (raises otherwise) - that part's not on documentation/good behavior alone.

But your same-host scenario (qli + qli-staging on one Artifactory instance) exposed a real bug, and it's worse than "undefined": I set up two preference stanzas pinning the same origin at different priorities, and APT doesn't take the higher priority or the latest one - it takes whichever stanza it registers first (filename order in preferences.d) and silently ignores the other. So one of the two priorities a user asked for just gets dropped, with no warning, and the outcome depends on file ordering rather than intent. That's a real gap in hostname-only pinning, not a corner case worth hand-waving away.

2. Sounds like we're aligned - I'll open a follow-up issue against qli-ci once this lands.

3. Since CLO turns out to be on its own instance (artifacts.codelinaro.org), separate from qsc-deb-releases, hostname pinning is actually precise for the Provide APT pinning option along with extra-repo #48 scenario specifically - it won't over-match anything else. The same-host ambiguity from (1) is a different, separate case from the one this PR is fixing.

4. Given (1), I think you're right that trying to extend --extra-repo-priority to handle disambiguation (suite, component, whatever) is the wrong direction - better to let the caller specify the actual Pin: criteria themselves via a passthrough, rather than us guessing a matching strategy per repo. I'll rework towards that.

On why Docker at all, given sbuild already isolates via unshare: the containers get built once and pushed to GHCR specifically so PR builds for the Ubuntu suites can docker pull a ready-made image on a GitHub-hosted runner and start building in under a minute, instead of running mmdebstrap to construct a chroot from scratch on an ephemeral runner every time. That's the main reason for the extra layer over bare sbuild - happy to discuss further if you think that trade-off isn't worth it, but that's a bigger conversation than this PR.

[Loïc Minier (lool)]

On why Docker at all, given sbuild already isolates via unshare: the containers get built once and pushed to GHCR specifically so PR builds for the Ubuntu suites can docker pull a ready-made image on a GitHub-hosted runner and start building in under a minute, instead of running mmdebstrap to construct a chroot from scratch on an ephemeral runner every time. That's the main reason for the extra layer over bare sbuild - happy to discuss further if you think that trade-off isn't worth it, but that's a bigger conversation than this PR.

A long while ago, I was running debootstrap as part of qcom-deb-images CI and it was running in about 30s on GH runners; I guess it might take a couple of minutes to setup a debian docker container with sbuild and an sbuild chroot in it, which compared to the duration of the build might or might be a big deal, don't know.

[Simon Beaudoin (simonbeaudoin0935)]

Loïc Minier (@lool)

I just tackled the issue above about same host + conflicting priorities. Now it errors out immediately with a clear message, instead of silently picking whichever preferences file APT happens to read first.

To be clear, this closes the silent failure, not the underlying limitation. You still can't pin qli and qli-staging differently; it'll just refuse loudly instead of guessing. I didn't try to smarten the matching (suite/Origin) since neither turned out reliably distinctive enough to trust. Real scenario needs to be addressed separately in the new issue I linked

[Loïc Minier (lool)]

I've only reviewed the code and doc changes, not the tests, I assume they were generated by the same LLM as the code