[Cleanup] Split implementation part off of certification specification

[mbuechse]

The Tests directory is itself a mixed bag.

• It contains test scripts for each standard that we want to include into certification.
• It contains the specs of our certificate scopes (the scs-compatible-?aas.yaml files).
• It contains the machinery for executing test runs (which in turn uses the former two items).

My feeling is that we should split the implementation part off of the yaml files.

For instance, it's clear that each module in the yaml corresponds to a version of a standard (basically one md file in the Standards folder), and each version of the scope lists the modules it consists of. To me, this is the essential certification specification.

Everything else -- the list of testcases and check scripts -- is the implementation, and I would split that off.

The specification should stay in the same repo as the Standards folder, and the implementation should go along with the Tests folder, wherever that may go.

[mbuechse]

Kind request for comments @fkr, @depressiveRobot, @jklare, @garloff

Here some example to make it more palpable. Have a look at https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compatible-iaas.yaml (abridged):

name: SCS-compatible IaaS
uuid: 50393e6f-2ae1-4c5c-a62c-3b75f2abef3f
url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-iaas.yaml
variables:
  - os_cloud
scripts:
- executable: ./iaas/openstack_test.py
  args: -c {os_cloud} {testcases}
  testcases:
  - id: scs-0100-syntax-check
    description: Flavor names starting `SCS-` comply with syntax.
    url: https://docs.scs.community/standards/scs-0100-w1-flavor-naming-implementation-testing#automated-tests
...
modules:
  ...
  - id: scs-0100-v3.1
    name: Flavor naming v3.1
    url: https://docs.scs.community/standards/scs-0100-v3-flavor-naming
    targets:
      main:
      - scs-0100-syntax-check
      - scs-0100-semantics-check
...
timeline:
  - date: 2025-09-09
    versions:
      next: draft
      v5.1: effective
      v4: deprecated
      v3: deprecated
...
versions:
  - version: next
    include:
      - opc-v2022.11
      - scs-0100-v3.1
      - scs-0101-v1
      - scs-0102-v2  # instead of scs-0102-v1
      - scs-0103-v1
      - ref: scs-0104-v1-2
        parameters:
          image_spec: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/iaas/scs-0104-v1-images-v5.yaml
      - scs-0114-v1
      - scs-0115-v1
      - scs-0116-v1
      - scs-0117-v1
      - scs-0121-v1
      - scs-0123-v2
      - scs-0302-v1

Move the top-level fields variables and scripts out of here, as well as the field targets from each module. What remains is the basic spec. And what was just removed is the implementation, which should go someplace else, along with the Tests folder.

[mbuechse]

Alternatively, we could just do regular human-readable markdown files for the high-level spec of the certificate scopes.

The Standards directory is itself a mixed bag, because the supplements follow a different regime than the normative documents (standards mainly, and decision records), and currently, the supplements mainly explain the tests that are implemented in the test suite, down to (redundantly) listing the testcases that are implemented. I don't know. Maybe it could be improved.

[mbuechse]

I think I was wrong in my earlier comment. The testcases are used to partition the tests according to frequency (some things are tested daily, other things less often), which is very essential for the specification. So the testcases need to be part of the specification. So we only have to extract the scripts and the variables and keep the rest.

[mbuechse]

I correct myself yet again. It's actually a matter of taste. Multiple options are possible:

• status quo: use yaml file to specify how often each property as to be tested -- but there, it's hidden in plain sight: who wants to dig through these yaml files?
• use testing notes to specify (normatively) how often a certain property has to be tested in prose (mainly for exceptions to the general rule), treat yaml file as implementation for the nerds

For instance, we currently say the Sonobuoy tests may be run monthly. Do you know where to look for this info?

[mbuechse]

So #1199 is my proposal now. I took the liberty of taking the idea to the extreme and show it off in a fork of this repo.