Add wrap runner run.sh with witness

[Uk-jake]

Summary

This PR wraps the GitHub Actions runner with witness run, so that the VM produces in-toto attestation.

This PR only covers invoking witness on the VM. This is a follow-up to #639.

Changes

Only the last command in runOnMachine() (cloudrunners/oci/main.go) is changed.
The command now does the following in order.

1. Sets up the runner environment (unchanged from the command).
2. Creates /tmp/witness for witness files and /home/ubuntu/_work as the witness working directory.
3. Generates signing key for the attestation.
4. Generates CA for network tracing and adds it to the system trust store.
5. Runs witness run as root.
6. Starts run.sh as ubuntu under witness, as before.

The resulting process tree

witness run (root)
└─ sudo -u ubuntu
   └─ bash -x run.sh
      └─ Runner.Listener
         └─ Runner.Worker
            └─ job steps

The attestation is stored at /tmp/witness/attestation.json on the VM.

Questions

1. Could you check the kernel version of the OCI image?
The network trace requires kernel 6.12 or later. The kernel version of the current OCI image has not been confirmed.

2. Is there another way to call the witness binary on the VM from this file?
In the current code, witness wraps the whole job, including the runner agent and all steps. However, we only want to monitor the job steps inside run.sh. If you have any other ideas, please let me know. Also, wrapping it this way will likely include ACTIONS_RUNNER_INPUT_JITCONFIG in the attestation.

3. Could you also review where the attestation should be saved?
The attestation will be used as input for the sbomit binary. This PR only changes the first step of the SBOMit pipeline (calling witness), but later we will need the path to this attestation file when running sbomit. In addition, it would be great to discuss how we plan to collect the final SBOM file once it is generated.

---

/kind enhancement
/area ci
/priority medium
/status needs-review

[koksay]

@Uk-jake I've actually worked on this yesterday; here are my changes: https://github.com/cncf/automation/compare/sbomit-integration?expand=1

The problem is, when we have it this way, because of the wrapping I assume, the communication between github and the runner is lost. In my case, the flow actually runs and uploads attestations and SBOM files to the S3 bucket; however, the job is not reflected on the GitHub Actions page.

2026/09/10 11:46:00 main.go:382: running ssh command command openssl genpkey -algorithm ed25519 -out signing-key.pem && chmod 600 signing-key.pem
2026/09/10 11:46:00 main.go:392: command succeeded command openssl genpkey -algorithm ed25519 -out signing-key.pem && chmod 600 signing-key.pem output
2026/09/10 11:46:00 main.go:382: running ssh command command sudo -E HOME=$HOME PATH=$PATH witness run --experimental --step ca-bootstrap -a network-trace --signer-file-key-path ./signing-key.pem -o /dev/null -- /bin/true
2026/09/10 11:46:14 main.go:392: command succeeded command sudo -E HOME=$HOME PATH=$PATH witness run --experimental --step ca-bootstrap -a network-trace --signer-file-key-path ./signing-key.pem -o /dev/null -- /bin/true output level=info msg="Starting prematerial attestors stage..."
level=info msg="Completed prematerial attestors stage..."
level=info msg="Starting material attestors stage..."
level=info msg="Starting material attestor..."
level=info msg="Finished material attestor... (9.55470705s)"
level=info msg="Completed material attestors stage..."
level=info msg="Starting execute attestors stage..."
level=info msg="Starting network-trace attestor..."
level=info msg="Starting command-run attestor..."
level=info msg="Generating new CA certificate (ECDSA P-256)"
level=info msg="Saved CA certificate to ./witness_nettrace_proxy/ca_cert.pem (0644) and key to ./witness_nettrace_proxy/ca_key.pem (0600)"
level=info msg="TCP proxy listening on [::1]:8888 (IPv6)"
level=info msg="TCP proxy listening on 127.0.0.1:8888 (IPv4)"
level=info msg="Running PreExec hooks"
level=info msg="Entering hooks-only mode"
level=info msg="Waiting for process exit to run PreExit hooks"
level=info msg="Last thread pausing for exit. Running PreExit hooks."
level=info msg="TCP proxy shutting down"
level=info msg="TCP proxy shutdown complete: all recordings finished"
level=info msg="Finished command-run attestor... (0.155142792s)"
level=info msg="Finished network-trace attestor... (0.455465239s)"
level=info msg="Completed execute attestors stage..."
level=info msg="Starting product attestors stage..."
level=info msg="Starting product attestor..."
level=info msg="Finished product attestor... (1.8402822890000001s)"
level=info msg="Completed product attestors stage..."
level=info msg="Starting postproduct attestors stage..."
level=info msg="Completed postproduct attestors stage..."

2026/09/10 11:46:14 main.go:382: running ssh command command sudo cp ./witness_nettrace_proxy/ca_cert.pem /usr/local/share/ca-certificates/witness-nettrace.crt && sudo update-ca-certificates
2026/09/10 11:46:15 main.go:392: command succeeded command sudo cp ./witness_nettrace_proxy/ca_cert.pem /usr/local/share/ca-certificates/witness-nettrace.crt && sudo update-ca-certificates output Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
/etc/ssl/certs/adoptium/cacerts successfully populated.
done.

2026/09/10 11:46:15 main.go:382: running ssh command command sudo -E PATH=$PATH:/home/ubuntu/.local/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.rustup/bin HOME=/home/ubuntu NVM_DIR=/home/ubuntu/.nvm witness run --experimental --step build -a network-trace --trace --attestor-command-run-trace-backend ebpf --signer-file-key-path ./signing-key.pem -o attestation.json -- bash -x /home/ubuntu/run.sh --jitconfig "${ACTIONS_RUNNER_INPUT_JITCONFIG}"
2026/09/10 11:46:20 main.go:392: command succeeded command sudo -E PATH=$PATH:/home/ubuntu/.local/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.rustup/bin HOME=/home/ubuntu NVM_DIR=/home/ubuntu/.nvm witness run --experimental --step build -a network-trace --trace --attestor-command-run-trace-backend ebpf --signer-file-key-path ./signing-key.pem -o attestation.json -- bash -x /home/ubuntu/run.sh --jitconfig "${ACTIONS_RUNNER_INPUT_JITCONFIG}" output level=info msg="Starting prematerial attestors stage..."
level=info msg="Completed prematerial attestors stage..."
level=info msg="Starting material attestors stage..."
level=info msg="Starting material attestor..."
level=info msg="Finished material attestor... (1.8442816990000002s)"
level=info msg="Completed material attestors stage..."
level=info msg="Starting execute attestors stage..."
level=info msg="Starting network-trace attestor..."
level=info msg="Starting command-run attestor..."
level=info msg="Loaded existing CA certificate"
level=info msg="TCP proxy listening on [::1]:8888 (IPv6)"
level=info msg="TCP proxy listening on 127.0.0.1:8888 (IPv4)"
level=info msg="Using tracer: ebpf for command-run"
level=info msg="Running PreExec hooks"
level=info msg="Entering hooks-only mode"
level=info msg="Waiting for process exit to run PreExit hooks"
+ SOURCE=/home/ubuntu/run.sh
+ '[' -h /home/ubuntu/run.sh ']'
+++ dirname /home/ubuntu/run.sh
++ cd -P /home/ubuntu
++ pwd
+ DIR=/home/ubuntu
+ [[ ! -z '' ]]
+ [[ -z '' ]]
+ run --jitconfig xxx
+ :
+ cp -f /home/ubuntu/run-helper.sh.template /home/ubuntu/run-helper.sh
Must not run interactively with sudo
Exiting runner...
+ /home/ubuntu/run-helper.sh --jitconfig xxx
level=info msg="Last thread pausing for exit. Running PreExit hooks."
level=info msg="TCP proxy shutting down"
level=info msg="TCP proxy shutdown complete: all recordings finished"
+ returnCode=1
+ [[ 1 -eq 2 ]]
+ [[ '' == \1 ]]
+ echo 'Exiting runner...'
+ exit 0
level=info msg="Finished network-trace attestor... (0.711004817s)"
level=info msg="Finished command-run attestor... (0.762236893s)"
level=info msg="Completed execute attestors stage..."
level=info msg="Starting product attestors stage..."
level=info msg="Starting product attestor..."
level=info msg="Finished product attestor... (1.838713837s)"
level=info msg="Completed product attestors stage..."
level=info msg="Starting postproduct attestors stage..."
level=info msg="Completed postproduct attestors stage..."

2026/09/10 11:46:20 main.go:382: running ssh command command sbomit generate attestation.json --format spdx23 --name my-sbom --output sbom.spdx.json
2026/09/10 11:46:30 main.go:392: command succeeded command sbomit generate attestation.json --format spdx23 --name my-sbom --output sbom.spdx.json output Parsed attestations (4 total): command-run=1, material=1, network-trace=1, product=1
SBOM written to sbom.spdx.json

2026/09/10 11:46:43 artifacts.go:66: uploaded artifact: /home/ubuntu/attestation.json -> oci://axtwf1hkrwcy/sbomit-test-bucket/gha-runner/local-test-runner-x86-ch2vc-runner-6w52z/20260910T114631Z/attestation.json (2499540 bytes)
2026/09/10 11:47:34 artifacts.go:66: uploaded artifact: /home/ubuntu/sbom.spdx.json -> oci://axtwf1hkrwcy/sbomit-test-bucket/gha-runner/local-test-runner-x86-ch2vc-runner-6w52z/20260910T114631Z/sbom.spdx.json (8728086 bytes)
2026/09/10 11:47:34 main.go:203: cleaning up: delete machine 163.192.29.182

This doesn't look promising; I'm looking into it.

[Uk-jake]

@koksay
I'll take a look as well. Thank you so much🙏

[Vyom-Yadav]

@koksay I think the issue here is:

Must not run interactively with sudo as witness currently requires sudo for working with ebpf programs. This is coming from: https://github.com/actions/runner/blob/759385a3510197a58b5c08dc1f373b74b9f4643b/src/Misc/layoutroot/run-helper.sh.template#L3-L8

As for why this is not streamed to GitHub actions ledger, because the listener is activated after this step of sudo check. As the sudo check fails, listener is never activated.

https://github.com/actions/runner/blob/759385a3510197a58b5c08dc1f373b74b9f4643b/src/Misc/layoutroot/run-helper.sh.template#L35-L37 (listener is started after the sudo check)

Can you try something like this once:

sudo -E \
  HOME=/home/ubuntu \
  PATH="$PATH:/home/ubuntu/.local/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.rustup/bin" \
  NVM_DIR=/home/ubuntu/.nvm \
  witness run \
    --experimental \
    --step build \
    -a network-trace \
    --trace \
    --attestor-command-run-trace-backend ebpf \
    --signer-file-key-path ./signing-key.pem \
    -o attestation.json \
    -- \
    setpriv \
      --reuid=ubuntu \
      --regid=ubuntu \
      --init-groups \
      env \
        -u SUDO_UID \
        -u SUDO_GID \
        -u SUDO_USER \
        HOME=/home/ubuntu \
        USER=ubuntu \
        LOGNAME=ubuntu \
        NVM_DIR=/home/ubuntu/.nvm \
        RUNNER_MANUALLY_TRAP_SIG=1 \
        PATH="$PATH:/home/ubuntu/.local/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.rustup/bin" \
        bash /home/ubuntu/run.sh \
          --jitconfig "${ACTIONS_RUNNER_INPUT_JITCONFIG}"

RUNNER_MANUALLY_TRAP_SIG would propagate this is a failure.

---

I have one quick question. During this integration, there might be a few bugs too iron out, how do we invoke this setup? Locally setting up things might not reproduce the prod env, is there any way you suggest?

[stupendoussuperpowers]

Adding to this, a few questions:

witness run (root)
└─ sudo -u ubuntu
   └─ bash -x run.sh
      └─ Runner.Listener
         └─ Runner.Worker
            └─ job steps

Is the only possible way to "wrap" the build workflow to wrap around the run.sh script? Is it possible to wrap a narrower part of a workflow job? My concern is that our attestations might contain a lot of "noise" which never resolve to package identities, and will be common/redundant across every single invocation (across projects).

As Vyom points out, the issue seems to be that of running the thing as sudo. Which way around this would be more amenable? To configure a specific user to have the required permissions or the witness explicitly dropping sudo permissions?

Also: +1 to Vyom's question on replicating this locally as that would speed up testing on our end.

[koksay]

I will close and reopen to trigger newly added testing

[koksay]

/hold

[koksay]

/retest

[github-actions]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign idvoretskyi after the PR has been reviewed.
You can assign the PR to them by writing /assign @idvoretskyi in a comment when ready.

Needs approval from an approver in each of these files:

• ci/cloudrunners/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[koksay]

/retest

[koksay]

/retest

[mfahlandt]

One addition here, is it possible to change the default SBOM generator to https://github.com/kusari-oss/waybill otherwhise comparing quality sizing will be not accurate