test(system_test): add the catalog_build SVT theme

test/system_test/README.md

@@ -83,12 +83,54 @@ tests of the stage are skipped (they need what it set up).
 The tests are data in [`library/vars/test_case_vars.py`](library/vars/test_case_vars.py);
 `svt/test_svt.py` generates one pytest test per entry.
 
+## Theme catalog_build
+
+For every catalog, one at a time: `./omnia.sh --select-catalog <catalog>`, then the
+repo_manager and image_build_manager FVT steps (no orchestrator deploy, no PXE boot).
+User guide: [`docs/CATALOG_BUILD.md`](docs/CATALOG_BUILD.md). Settings:
+[`catalog_build_test_config.yml`](catalog_build_test_config.yml); design:
+[`catalog_build_design.md`](catalog_build_design.md).
+
+```bash
+./setup_env.sh --theme catalog_build            # configure (freezes the catalog plan) -> install -> creds
+./setup_env.sh --theme catalog_build --force    # cleanup first (deletes OMNIA_DATA_PATH)
+./run_validation.sh svt_catalog_build list      # the catalog stages and their tests
+./run_validation.sh svt_catalog_build all       # every catalog stage, then summary
+./run_validation.sh svt_catalog_build all --start-stage catalog_02
+./run_validation.sh svt_catalog_build all --retry
+```
+
+`--configure` resolves `catalog_build.catalogs` (`all` = the `./omnia.sh
+--list-catalogs` order) into the plan of the run, `reports/catalog_build_plan.json`.
+The theme's stages (`catalog_01`, `catalog_02` ...) and tests are generated from it
+([`library/vars/catalog_vars.py`](library/vars/catalog_vars.py)) and run like the
+build_stream tests. IDs are `SVT_CATALOG_BUILD_<catalog NN><step NN>`.
+
+| Test of a catalog stage | What it does | Step |
+|---|---|---|
+| `NN00` | Put back the shared runtime inputs, apply `catalog_overrides/<catalog>/<domain>/<file>`, write `repo_config`, `caching_policy`, `repo_policies`, `force_rebuild` and `functional_groups_source: catalog` into `<OMNIA_DATA_PATH>/<domain>/input/<project>/`, then `./omnia.sh --select-catalog` | `svt select <catalog>` |
+| `NN01`... | Every `steps` entry `[domain, tag]` of the theme block, as two tests | `fvt_<domain> <tag> exec`, `fvt_<domain> <tag> verify` |
+| last | `cleanup_between` (`images`, `repos`, `full`); runs even after a failure | `fvt_<domain> cleanup... test` |
+
+A catalog with an aarch64 layer is one skipped test (`NN00`, with the reason) unless
+`include_aarch64: true` (needs `aarch64_inventory_host_ip` in user_inputs.yml). With
+`continue_on_failure: true` every catalog stage is a soft stage: a failed catalog does
+not stop the next one; the summary still fails. After a failed `exec` the orphaned
+`ansible-playbook <domain>.yml` workers (and a running `dnf download`) are stopped, so
+a playbook killed at the FVT timeout cannot overlap the next test.
+
+From GitLab (`svt.theme: "catalog_build"` in `pipeline/svt_pipeline_config.yml`) every
+catalog is its own pipeline job, `catalog_01` ...; see
+[docs/PIPELINE_CONFIG.md](docs/PIPELINE_CONFIG.md#theme-catalog_build-one-job-per-catalog).
+How the theme compares with build_stream: [docs/CATALOG_BUILD.md](docs/CATALOG_BUILD.md).
+
 ## Documentation
 
 | Doc | Contents |
 |---|---|
 | [docs/SETUP.md](docs/SETUP.md) | `./setup_env.sh` (configure, cleanup incl. NFS shares, install, credentials, status), `user_inputs.yml`, the dataset generator |
 | [docs/TEST_CONFIG.md](docs/TEST_CONFIG.md) | Every setting of `build_stream_test_config.yml`: catalogs, the cleanup flags per suite (telemetry, NFS shares ...), the report |
+| [docs/CATALOG_BUILD.md](docs/CATALOG_BUILD.md) | **User guide of the `catalog_build` theme**: prerequisites, the two files to edit, step by step, every setting, reading the report, failures, the GitLab pipeline (one job per catalog), how it compares with build_stream, troubleshooting |
 | [docs/RUNNER.md](docs/RUNNER.md) | `./run_validation.sh`, the run and its guard, retry, the SVT report, adding a theme, stage or test |
 | [docs/OPENBAO.md](docs/OPENBAO.md) | Passwords from OpenBao: one place for its settings, the pipeline and manual ways to read them |
 | [docs/PIPELINE.md](docs/PIPELINE.md) | Run the SVT from GitLab: how it works, where each secret goes, setup, **updating an existing pipeline**, troubleshooting |

test/system_test/_setup.py

@@ -49,6 +49,7 @@
 from typing import Any, Callable, Dict, List, Optional
 
 from library.functions import (
+    catalog_func,
     cleanup_func,
     readiness_func,
     run_func,
@@ -172,9 +173,23 @@ def _mark_run(name: str, outcome: Dict[str, Any]) -> None:
             _emit(message)
 
 
+def _configure(config: Dict[str, Any]) -> Dict[str, Any]:
+    """Configure; a catalog theme then freezes its catalog plan for the new run.
+
+    The plan (catalog_func.write_plan) is what the theme's stages and tests are
+    generated from; when it cannot be built, configure fails and says why.
+    """
+    outcome = setup_func.configure(config, _emit)
+    if outcome["status"] != "passed" or not THEMES[config["theme"]].get("catalog_plan"):
+        return outcome
+    rows, error = catalog_func.write_plan(config["catalog"], outcome["run_id"], _emit)
+    outcome["rows"] = [*(outcome.get("rows") or []), *rows]
+    return {**outcome, "status": "failed", "error": error} if error else outcome
+
+
 def _steps(args: argparse.Namespace, config: Dict[str, Any]) -> List[tuple]:
     """Return the (name, step) list of the requested action."""
-    configure = ("configure", lambda: setup_func.configure(config, _emit))
+    configure = ("configure", lambda: _configure(config))
     cleanup = ("cleanup", lambda: cleanup_func.cleanup(config, _emit, args.yes))
     install = ("install", lambda: setup_func.install(config, _emit, args.force, args.yes))
     if args.action == "all":

test/system_test/catalog_build_design.md

@@ -0,0 +1,136 @@
+# SVT Theme `catalog_build` — Design
+
+Status: **Implemented** on the merged SVT framework (PR #5711)
+Runner category: `svt_catalog_build`
+
+## 1. Goal
+
+A manual run on one OIM (no GitLab needed). For **every catalog**, one at a time:
+
+```
+select catalog -> repo_manager FVTs (exec + verify) -> image_build_manager FVTs (exec + verify)
+```
+
+Every step is the domain's own FVT run (`test/<domain>/_run.py fvt_<domain> <tag>
+exec|verify`); the theme adds no test logic. Scope ends after the image build: no
+orchestrator deploy, no PXE boot.
+
+```bash
+./setup_env.sh --theme catalog_build             # configure (plan) -> install -> creds
+./run_validation.sh svt_catalog_build list       # catalog stages and their tests
+./run_validation.sh svt_catalog_build all        # every catalog stage, then summary
+./run_validation.sh svt_catalog_build all --start-stage catalog_02
+./run_validation.sh svt_catalog_build all --retry
+```
+
+## 2. Aligned with the build_stream theme
+
+A theme of the merged framework is **data**: its stages and `stage_info` in
+`theme_vars.THEMES`, its tests in `test_case_vars.TEST_CASES` (ID, title, order,
+stage, FVT step), run by the generic `svt/test_svt.py` through
+`stage_func.run_tests` (pytest `-m "<theme> and <stage>"`). catalog_build is the
+same; only its data is generated:
+
+| Part | build_stream | catalog_build |
+|---|---|---|
+| Theme entry (`theme_vars.py`) | fixed stages | stages, `soft_stages`, `stage_info` from the catalog plan |
+| Tests (`test_case_vars.py`) | written by hand | generated from the plan (`catalog_vars.catalog_test_cases`) |
+| Test runner | `svt/test_svt.py` -> `run_svt_test` | the same |
+| Report (JSON/HTML) | `svt_report_func` / `svt_report_html` | the same (stage titles name the catalog) |
+| Failure handling | soft stages, skip the rest of a stage, `--retry`, `--start-stage` | the same (`continue_on_failure` -> soft stages) |
+| Setup (`setup_env.sh`) | configure, cleanup, install, creds | the same; configure also writes the plan |
+| Cleanup steps | build_stream, telemetry, orchestrator, image, repo, main | the same steps and flags |
+| Credential helper | `test/build_stream/setup_env.sh` | the same (`creds_suite: build_stream`), repo_manager and image_build_manager only |
+| Input profile | 9 input files | `omnia.env`, `repo_manager_config.yml`, `image_build_config.yml` |
+
+## 3. The catalog plan
+
+The catalog list lives in `catalog_build_test_config.yml`, but the stages and tests
+must exist when the vars modules are imported (`domain_vars`, pytest markers, the
+pipeline generator). So `./setup_env.sh --configure` (the `_configure` step of
+`_setup.py`) resolves the list and freezes it:
+
+1. `catalogs: all` = every `./omnia.sh --list-catalogs` entry in that order; a list
+   keeps its own order (names or numbers, `setup_func.resolve_catalog`).
+2. A catalog with an aarch64 functional layer is marked skipped unless
+   `include_aarch64: true`; `true` needs `aarch64_inventory_host_ip` in
+   user_inputs.yml, else configure fails.
+3. The plan is written to `reports/catalog_build_plan.json` with the run id and the
+   normalized settings, and printed.
+
+`catalog_vars.load_plan()` reads it at import time. A changed
+`catalog_build_test_config.yml` already needs a new `--configure` (the run guard),
+so the plan always belongs to the current run; `initialization` checks it
+(`Catalog plan of this run`). Without a plan the theme has no catalog stage.
+
+## 4. Tests of a catalog stage
+
+IDs `SVT_CATALOG_BUILD_<catalog NN><step NN>`, order `2000 + NN*100 + step`:
+
+| Test | What it does | Step |
+|---|---|---|
+| `NN00 select` | SVT action (`catalog_func.run_action`): put back the shared runtime inputs, apply `catalog_overrides/<catalog>/<domain>/<file>`, write the global settings into `<OMNIA_DATA_PATH>/<domain>/input/<project>/`, then `./omnia.sh --select-catalog` | `svt select <catalog>` |
+| `NN00 skip` | Instead of all the others, for a skipped catalog: one skipped test with the reason | `svt skip <catalog>` |
+| `NN01`... | For every `steps` entry `[domain, tag]`: `<tag> exec` (the playbook), then `<tag> verify` | `fvt_<domain> <tag> exec` / `verify` |
+| last | `cleanup_between` steps (`always`: they run even after a failure) | `fvt_<domain> cleanup... test` |
+
+Global settings written before every catalog: `repo_config`, `caching_policy`,
+`repo_policies` (`"<version>/<arch>/<repo>": {policy, caching}`) into
+`repo_manager_config.yml`; `build_image.force_rebuild` (default **true**) and
+`functional_groups_source: catalog` into `image_build_config.yml`.
+
+The first test of a stage is the repo_manager precheck after select, so the existing
+subscription-aware repository validation runs for every catalog; the theme adds no
+repository check of its own. The FVT playbook timeout stays the omnia_auto default
+(7200 s).
+
+## 5. Hooks in shared code (build_stream unchanged)
+
+| File | Change |
+|---|---|
+| `library/functions/test_func.py` | Optional test-case fields: `action` (an SVT action instead of an FVT step), `always` (runs after an earlier failure of the stage), `stop_leftovers` (after a failed exec, stop the orphaned `ansible-playbook <domain>.yml` workers and their children, which run in their own session and survive the FVT timeout). build_stream's tests carry none of them. |
+| `library/functions/fvt_func.py` | `describe_step` labels an SVT action `svt <action> ...` |
+| `library/functions/config_func.py` | A theme with `catalog_plan` loads its block with `catalog_settings.py` |
+| `library/functions/readiness_func.py` | `Catalog plan of this run` check for catalog themes |
+| `library/functions/setup_func.py` | `bundled_catalogs` and `select_catalog` made public (reused by the plan and the select action) |
+| `_setup.py` | `_configure`: configure, then the plan for catalog themes |
+| `library/vars/theme_vars.py`, `test_case_vars.py` | The catalog_build entry and its generated tests |
+| `pipeline/setup_svt_project.py`, `pipeline/svt_ci.py` | One job per catalog, and the plan check at initialization (section 7) |
+
+New files: `library/vars/catalog_vars.py`, `library/functions/catalog_settings.py`,
+`library/functions/catalog_func.py`, `catalog_build_test_config.yml`,
+`dataset_generator/profiles/catalog_build.yml`, `ut/test_catalog_build.py`.
+
+## 6. Settings (`catalog_build_test_config.yml`)
+
+| Key | Default | Effect |
+|---|---|---|
+| `catalogs` | `all` | `all` or an ordered list of `--list-catalogs` names / numbers |
+| `include_aarch64` | `false` | `false` skips catalogs with an aarch64 layer |
+| `repo_config` | `partial` | `always` / `partial` |
+| `caching_policy` | `true` | repo_manager caching |
+| `repo_policies` | `{}` | per-repository policy (`always` / `partial` / `never`) and caching |
+| `force_rebuild` | `true` | `true` rebuilds every image; `false` reuses a matching image |
+| `continue_on_failure` | `true` | every catalog stage is soft |
+| `cleanup_between` | `none` | `none` / `images` / `repos` / `full` after every catalog |
+| `steps` | 8 steps | `[domain, tag]` FVT steps of every catalog |
+
+`cleanup` and `report` blocks: as in `build_stream_test_config.yml`.
+
+## 7. GitLab pipeline
+
+The SVT pipeline (`pipeline/`) creates one job per theme stage when the CI files are
+rendered, before any `--configure` on the OIM. For a catalog theme,
+`setup_svt_project.catalog_plan()` therefore resolves the plan on the generating host
+with the same `catalog_func.plan()` (this checkout's `catalog_build_test_config.yml`,
+`user_inputs.yml` and `./omnia.sh --list-catalogs`), and renders:
+
+* one job per catalog (`catalog_01` ...), titled with the catalog, `allow_failure`
+  when `continue_on_failure` is true, timeout `timeouts_hours.catalog_stage`;
+* `SVT_CATALOG_PLAN` = `<stage>=<catalog>,...`.
+
+`svt_ci.py initialization` compares `SVT_CATALOG_PLAN` with the plan `--configure`
+wrote on the OIM, before cleanup and install, and fails when they differ (a catalog list
+edited in the project, or a tested branch with other bundled catalogs), because those
+catalogs would have no job. The fix is `./setup_svt_project.py --update --update-inputs`.
+build_stream's CI files are unchanged (no `SVT_CATALOG_PLAN`).

test/system_test/catalog_build_test_config.yml

@@ -0,0 +1,128 @@
+# Copyright 2026 Dell Inc. or its subsidiaries. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+---
+# =============================================================================
+# catalog_build_test_config.yml - settings of the SVT theme catalog_build
+# =============================================================================
+# Read by ./setup_env.sh --theme catalog_build and by ./run_validation.sh
+# svt_catalog_build. The cluster values (IPs, names, repository URLs) are in
+# user_inputs.yml; credentials are never written here
+# (./setup_env.sh --theme catalog_build --set-domain-creds).
+# ./setup_env.sh --configure resolves the catalog list below into the plan of the
+# run (reports/catalog_build_plan.json): one stage per catalog (catalog_01,
+# catalog_02 ...). Each stage selects its catalog (./omnia.sh --select-catalog),
+# then runs every step as the domain's FVT "<tag> exec" and "<tag> verify". A
+# change here needs ./setup_env.sh --theme catalog_build --configure (a new run).
+# =============================================================================
+
+# =============================================================================
+# CATALOG_BUILD  (the plan of the run; written into the runtime inputs before
+#                 every catalog, not into the suites' test_config.yml)
+# =============================================================================
+catalog_build:
+  # catalogs: all | a list of names or numbers of ./omnia.sh --list-catalogs.
+  #   all = every bundled catalog in the --list-catalogs order; a list runs in
+  #   the order given, e.g.
+  #     catalogs:
+  #       - rhel/10.2/slurm_x86_64_no_vast.json
+  #       - rhel/10.2/slurm_service_k8s_x86_64_no_vast.json
+  catalogs: all
+  # include_aarch64: false (default) skips every catalog with an aarch64 layer
+  #   (aarch64-only or mixed): its stage holds one skipped test with the reason.
+  #   true runs them; it needs
+  #   image_build_manager.image_build_config.aarch64_inventory_host_ip in
+  #   user_inputs.yml.
+  include_aarch64: false
+  # ---- global for the whole run: written into the runtime inputs
+  #      (<OMNIA_DATA_PATH>/<domain>/input/<project>/) before every catalog ----
+  # repo_config: always | partial   (repo_manager_config.yml repo_config)
+  repo_config: partial
+  # caching_policy: true | false   (repo_manager_config.yml caching_policy)
+  caching_policy: true
+  # repo_policies: per repository "<version>/<arch>/<repo>": {policy, caching}
+  #   policy: always | partial | never; caching: true | false, e.g.
+  #     repo_policies:
+  #       "10.2/x86_64/epel": {policy: always, caching: false}
+  repo_policies: {}
+  # force_rebuild: true (default) builds every image of every catalog from
+  #   scratch; false reuses an image group already built (image build skips it).
+  force_rebuild: true
+  # continue_on_failure: true makes every catalog stage soft: a failed catalog
+  #   does not stop the next one (the summary still fails). false stops the run
+  #   at the first failed catalog.
+  continue_on_failure: true
+  # cleanup_between: none | images | repos | full, after every catalog (runs
+  #   even when a step of the catalog failed):
+  #   images = fvt_image_build_manager cleanup_images test
+  #   repos  = fvt_repo_manager cleanup test (the next catalog prepares again)
+  #   full   = images + image_build_manager cleanup + repos
+  cleanup_between: none
+  # steps: the FVT steps of every catalog, in order: [domain, tag]
+  #   (domain repo_manager | image_build_manager; tag one of its FVT tags).
+  #   Each becomes two SVT tests: "<tag> exec" (the playbook) and "<tag> verify".
+  steps:
+    - [repo_manager, precheck]
+    - [repo_manager, prepare]
+    - [repo_manager, execute]
+    - [repo_manager, status]
+    - [image_build_manager, precheck]
+    - [image_build_manager, validate]
+    - [image_build_manager, prepare]
+    - [image_build_manager, build]
+
+# =============================================================================
+# CLEANUP  (./setup_env.sh --theme catalog_build --cleanup, or --force;
+#           destructive, asks 'yes' unless --yes)
+# =============================================================================
+# The same steps as the build_stream theme, so an earlier BuildStream, telemetry
+# or orchestrator deployment cannot block omnia.sh --cleanup --all: build_stream
+# (when installed), telemetry (when deployed), orchestrator, image_build_manager,
+# repo_manager, main. See build_stream_test_config.yml for every flag.
+cleanup:
+  build_stream:
+    enabled: true
+    postgres_backup: false
+    cleanup_credentials: true
+  telemetry:
+    enabled: true
+    delete_sinks_volume: true
+    cleanup_credentials: true
+    cleanup_logs: true
+  orchestrator:
+    enabled: true
+    cleanup_credentials: true
+    cleanup_slurm: true
+    cleanup_k8s: true
+    cleanup_benchmark_tools: true
+  image_build_manager:
+    enabled: true
+    cleanup_credentials: true
+  repo_manager:
+    enabled: true
+    cleanup_credentials: true
+  main:
+    enabled: true
+    cleanup_all: true
+
+# =============================================================================
+# REPORT  (one folder and one SVT report per run)
+# =============================================================================
+report:
+  # report_path: parent of the run folders (default: "" =
+  #   test/system_test/reports). Keep it outside OMNIA_DATA_PATH.
+  report_path: ""
+  # report_name: (default: "" = svt_catalog_build_report)
+  report_name: ""
+  # run_id: (default: "" = a new svt-<UTC time> at every ./setup_env.sh --configure)
+  run_id: ""