[elezar]

Summary

Add hardware-free CDI injection tests to the features suite and tmachine environments that provide the fixtures. Each test creates a sandbox that selects one scenario device through the Docker or Podman driver and checks what the workload observes against expectations recorded in a spec. This PR is no longer stacked on #4529.

Related Issue

Related to #4460. This PR supplies the initial CDI test infrastructure; the issue remains open to track follow-up coverage.

Changes

• Compose tmachine environments from base environments with a shared helper, and add a features table so a feature's setup playbooks and inputs apply to each base it supports. The Keycloak and Podman user-namespace variants are built the same way. The Ubuntu Docker Keycloak environment now also runs the base's test-tools playbook.
• Let an environment write driver settings to /etc/openshell-test/driver.toml. The binaries installer renders them into gateway.toml. The packaged installer writes the service user's operator gateway.toml only when settings exist.
• Add a cdi feature with environments ubuntu-docker-rootful-cdi, fedora-podman-rootful-cdi and fedora-podman-rootless-cdi. The cdi_fixtures role creates one CDI device per scenario of the tmachine.example/cdi-scenario class, installs the spec in /etc/cdi, publishes the expectations, and enables caller driver config, which the gateway requires to accept a per-sandbox CDI device.
• Keep the fixtures usable by containers on SELinux hosts. The fixture files live under /usr/local/share, whose default type containers can read, and the socket's directory is labeled container_file_t. A systemd service serves the echo socket with systemd-socket-activate --inetd. It opens the socket to any user, because the process creates it root-only, and runs as container_t on SELinux hosts, because a container may not connect to a socket owned by a service's default unconfined_service_t. The playbook connects to the socket after starting the service and fails if it does not echo. Another service creates the host node /dev/tmachine-cdi-null on every boot with the SELinux type of /dev/null, so a rootless runtime that bind-mounts it can use it.
• Add the cdi features package with a scenario wrapper (env, exists, contents, filetype, read, write, non-root, in-group-of and echo-socket checks and a rejection case) and 10 tests, one per scenario. echo-socket connects to a unix socket from the workload and expects its payload back; it needs perl in the workload image, which the default image has, and reports no-client without it. Run it from e2e-suites.yml in the three environments above, with the deb installer for Docker and the rpm installer for Podman.

Removed scenario: device-node-with-gid

This scenario is not in this PR. The Podman driver rejects a CDI-added supplementary group at sandbox startup (#5305), and the scenario is recorded there and below so it can be restored once the driver is fixed. Its original definition in the cdi_fixtures role spec (vars/scenarios.yaml), with a group openshell-cdi-fixture created in the guest and the node made with mknod:

device-node-with-gid scenario (verbatim from fb39ac1)

# The device is mode 0660 with a group that only `additionalGids` grants.
- name: device-node-with-gid
  fixtures:
    - {path: group-device, type: chardev, major: 1, minor: 3, mode: "0660", group: openshell-cdi-fixture}
  edits:
    additionalGids: ["{{ cdi_fixture_group.gid | int }}"]
    deviceNodes:
      - path: /dev/tmachine-cdi-group-null
        hostPath: "{{ cdi_scenario_dir }}/group-device"
        permissions: rw
        uid: 0
        gid: "{{ cdi_fixture_group.gid | int }}"
        fileMode: 432
  expect:
    checks:
      - {type: non-root}
      - {type: group-of, path: /dev/tmachine-cdi-group-null}
      - {type: write, path: /dev/tmachine-cdi-group-null, expect: denied}

The group-of check was later renamed in-group-of. The role's tasks/scenarios.yaml creates the system group openshell-cdi-fixture (registered as cdi_fixture_group) and makes the node with mknod ... c 1 3.

Release Notes

Testing

• Checks appropriate to the affected code and behavior pass: the repository pre-commit hooks pass on each commit, and Clippy for the features workspace is clean.
• Unit tests added/updated: 12 tests in the cdi package cover check parsing, comparison and the generated probe shell, including a probe against a real unix socket.
• E2E tests added/updated: the cdi package passed all 22 tests (the 12 unit tests and the 10 scenarios) in each of ubuntu-docker-rootful-cdi (deb), fedora-podman-rootful-cdi (rpm) and fedora-podman-rootless-cdi (rpm), with nix run .#openshell-test-runner -- --environment <environment> --installer <installer> --family features --package cdi on a local Apple Silicon Mac, using locally built arm64 artifacts.

The expectations in the spec record the sandbox's behavior today, so they will change as CDI grant derivation changes. x86_64 was not run locally. Rootless Podman cannot create device nodes, so the device-node scenario uses a host node at the same path as the container path.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Testing documentation updated; runtime architecture is unchanged

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4461.docs.buildwithfern.com/openshell

[github-actions]

Compatibility review (advisory)

Needs review

No source-level SDK, CLI, configuration, policy, Helm, or state contract changes were found. Qualification remains incomplete because release VM drivers embed newly hashed Linux runtime archives whose semantic delta is unavailable in the supplied source.

Not inspected: api-behavior

1. api-behavior — NOT ACKED

The pinned x86_64 and aarch64 Linux VM runtime archive hashes changed, but the archive contents and semantic differences are not present in the supplied source.

• Impact: Consumers of Linux VM sandboxes may receive changed libkrun, firmware, or umoci behavior that cannot be checked for compatibility from this repository.
• Action: Provide an immutable provenance or source diff for both replacement archives and confirm runtime compatibility, or restore the baseline archives for this train.
• PR: test(features): add CDI injection scenarios for Docker and Podman #4461

Review details

• Execution: incomplete
• Candidate: PR #4461 (782cf8bdcab7138914bfdd859ae6a3f904c4ead6)
• Baseline: 360f6d5cc2ad707bf84cbcda239e08086a527cac (360f6d5cc2ad707bf84cbcda239e08086a527cac)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[SDAChess]

Some questions, I'm surprised there are changes related to the VM driver, are these on purpose?

[SDAChess]

If there is no other way to check than to run some bash code then let's do that, but I would feel better about having these checks be Rust code.

[elezar]

We are trying to determine the state of the sandbox filesystem (after CDI injection and landlocked permissions). If you can think of a better way to do that, then I'd be happy to migrate.

[github-actions]

This pull request has had no activity for 14 days and is now marked stale.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

Label test:e2e applied, but pull-request/4461 is at fb39ac1 while the PR head is 3f0a69c. A maintainer needs to comment /ok to test 3f0a69cde952a3939965efdee9d5c3bacc6689ac to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

[elezar]

/ok-to-test 3f0a69c

[SDAChess]

Keeping the shell is fine with me. One small fix for the echo fixture and a non-blocking parser nit below.

[SDAChess]

Please add --inetd here. Without it, systemd-socket-activate passes the accepted connection on fd 3, while cat reads stdin, so this service does not actually echo. I reproduced this locally: the current invocation returned no data, and adding --inetd echoed the payload. The current scenario only checks the socket type and sibling-file absence, so the green test does not catch this. The systemd documentation also uses --inetd for its cat echo-server example.

[SDAChess]

Nit: consider rejecting a present, non-array checks value instead of treating it as an empty list. Accidentally writing a YAML mapping instead of a list would silently drop the scenario-specific checks while the common checks could still pass. A focused unit test for that malformed input would help protect against accidentally losing coverage. I'm happy to let this go in without this change.

[matthewgrossman]

Thanks for this, Evan. The layering (withSetup/features), one crate per feature, explicit CI rows, and expect_compute_driver all fit how we're structuring features suites. My comments are about how inputs reach the tests, not the scenarios themselves.

1. Scenario expectations should come through OPENSHELL_TEST_* and test-env.yaml.
The suite reads a hard-coded guest path (/var/lib/openshell-test-inputs/cdi/scenarios.json), with an override named OPENSHELL_CDI_SCENARIOS. The pattern from the Keycloak example is that setup includes playbooks/test-env.yaml to write OPENSHELL_TEST_* into /etc/openshell-test/env.d/<name>.env, the runner sources those into the test environment, and the suite reads plain variables. Reading a fixed guest path means the suite doesn't run unchanged against any deployment that supplies the same variables, and the openshell-test-inputs file-path style is the one we're moving away from (OPENSHELL_TEST_INPUT_DIR in the old Podman playbooks). Could the cdi_fixtures role publish something like OPENSHELL_TEST_CDI_SCENARIOS (and the device class, if useful) this way?

2. /etc/openshell-test/driver.toml is a new general mechanism.
It's slurped in detect.yaml and templated into the Docker and Podman gateway.toml. It fits "one environment per gateway config", but it isn't one of the channels we've settled on, and other features will likely start using it. Is this meant to be the shared way for an environment to add driver settings? If so I'd like to agree on the name/contract and document it in TESTING.md; if it's CDI-specific, it may be better scoped that way.

3. Packaged gateway override.
openshell_packaged_gateway now writes a fresh operator gateway.toml containing only the extra driver settings. Please confirm nothing the generated config previously supplied for the rpm Podman rootless path (socket path, openshell_gateway_podman_driver_settings, etc.) is dropped when this file exists. The PR notes validation was arm64-only, so CI on x86_64 will be the first real check.

4. Hidden perl dependency.
echo-socket needs perl in the workload image. Fine for the default image, but it'd be good for that to be explicit rather than inferred. For example, tie it to OPENSHELL_TEST_BASE_IMAGE, or state it in the suite docs.

Smaller notes, non-blocking:

• The write probe uses : >> path, so on a path that doesn't exist yet (.../directory/probe, .../new-file) it creates a file on the host if the write is allowed. The comment says nothing is written.
• rejected isn't exercised by any scenario after device-node-with-gid was dropped, and an empty rejected string matches any failure.
• The expectations record current behavior (e.g. read: denied on read-only mounts). A short "characterization, not spec" note would keep them from being read as desired behavior.