policy-test: use local registry by davidweisse · Pull Request #2583 · edgelesssys/contrast

davidweisse · 2026-08-10T13:37:33Z

This uses the --insecure-registry flag from #2580 to configure a local registry via crane registry serve during the policy test suite evaluation. To use this, first run just policy-registry and wait until it says Registry is ready.. Then run just policy. The first target will start a local registry via crane registry serve and push the needed images into it while also writing to the image replacements file.

One problem is: genpolicy also pulls the configured pause image (ghcr.io/edgelesssys/kuberenetes/pause:3.6 in our case), which cannot be replaced by the image replacements file, since it isn't present in any of the deployment YAMLs. We configure ghcr.io as an insecure registry and set up an HTTP proxy that reroutes requests to ghcr.io to the local registry. For this to work, we of course push the pause image to the local registry beforehand.

Doing some testing, I get the following results:

The first image pull will always take longer, as it has to pull the layers. About 2-3s to pull busybox from a local registry.
Subsequent pulls only need the manifest/config, this takes ~120ms.
Policy evaluation takes ~30-50ms for each RPC request.

Fixes CON-251

This PR is currently rebased on both #2580 and #2582, so only the most recent commit is relevant.

linear-code · 2026-08-10T13:37:37Z

CON-251

davidweisse · 2026-08-19T12:30:55Z

Update: Instead of the weird proxy setup, we can dynamically overwrite the pause image in the genpolicy settings file before running contrast generate. We can then explicitly pass that file to generate via a command line flag.

Also, with the rebase, the rego evaluation timing goes down to <10ms (~10-20ms max for CreateContainerRequest and <1ms for for others)

charludo

Thanks! I have a bunch of small comments, but generally, this LGTM. Tested it out locally and seems to work.

One thing I didn't catch on the last PR - maybe the policy-test should be moved from top-level into tools/...? Tools also doesn't feel quite right though. IDK.

I still don't really like having to run two terminals/just commands. crane uses https://github.com/google/go-containerregistry, so why not do this ourselves? Super rough sketch:

import (
  "net"
  "net/http"
  "github.com/google/go-containerregistry/pkg/registry"
  "github.com/google/go-containerregistry/pkg/crane"
  "github.com/google/go-containerregistry/pkg/v1/layout"
)

ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil { return fmt.Errorf("listen: %w", err) }
srv := &http.Server{Handler: registry.New(registry.Logger(log.New(io.Discard, "", 0)))}
go srv.Serve(ln)
defer srv.Close()
addr := ln.Addr().String()

for _, img := range []string{busyboxRef, pauseRef} {
  if err := crane.Copy(img, addr+"/"+strings.SplitN(img, "/", 2)[1], crane.Insecure); err != nil {...}
}

idx, err := layout.ImageIndexFromPath(initializerOCIDir)
crane.PushIndex(idx, addr+"/contrast/initializer:latest", crane.Insecure)

where initializerOCIDir comes from nix, e.g. through another install -D in assets/. We could even pull busybox and pause images in the nix build and persist them, so no re-download between runs. The only not-so-nice thing about this would be to have to either write the image-replacements from go, or (probably better), allow generatePolicies to optionally take a map of image replacements directly.

The second point should probably be decided on before implementing the review comments (@burgerdev), since going the go-containerregistry route would make a lot of them irrelevant.

charludo · 2026-08-24T09:07:21Z


+	// Patch the pause image in genpolicy-settings.json to use the insecure registry if specified.
+	if flags.insecureRegistry != "" {
+		genpolicySettings = bytes.ReplaceAll(genpolicySettings, []byte("ghcr.io/edgelesssys/kubernetes/pause"), []byte(flags.insecureRegistry+"/kubernetes/pause"))


This fails silently if no match is found. Maybe add a check to ensure the new pause image is written to the settings.

charludo · 2026-08-24T09:09:58Z

 default_deploy_target := "openssl"
 default_platform := "${default_platform}"
 default_set := "${set}"
+default_registry := "${container_registry}"


Is there a reason to add the new default_registry here, or could we just do

push target set=default_set registry=container_registry:

below?

charludo · 2026-08-24T09:11:46Z

+		"--settings=" + filepath.Join(workDir, "genpolicy-settings.json"),
 		"--genpolicy-cache-path=" + filepath.Join(workDir, "layers-cache.json"),
 		"--image-replacements=" + flags.imageReplacementsFile,
+		"--insecure-registry=" + flags.insecureRegistry,


I think cobra's StringArray turns the empty default into [""], so something is always passed here. Maybe

if flags.insecureRegistry != "" { args = append(args, "--insecure-registry="+flags.insecureRegistry) }

or similar?

charludo · 2026-08-24T09:19:00Z

+    tail -n 0 --pid $PID -f ./{{ workspace_dir }}/registry.log
+
 # Run the policy test suite.
-policy: initializer


Shouldn't we still push the initializer here? If it didn't change, it's close to a no-op, but pushing it only in policy-registry means needing to restart the registry when we made a change and want to re-test.

charludo · 2026-08-24T09:19:47Z

+    fi
+    mkdir -p ./{{ workspace_dir }}
+    echo "Setting up registry..."
+    crane registry serve --address localhost:5000 > ./{{ workspace_dir }}/registry.log 2>&1 &


Probably need to wait (shortly) for readiness

The short delay until crane starts serving has been a recurring pain point in Constellation's Bazel builds. If possible, I'd really prefer to run the registry from Go code, as @charludo suggested.

charludo · 2026-08-24T09:21:18Z

+    set -euo pipefail
+    if curl -sf localhost:5000/v2/ > /dev/null; then
+        echo "Registry already running on port 5000."
+        exit 1


This should be exit 0, we're basically saying "nothing to do, already up!".
Alternatively, exit 1 and "port 5000 in use, can't start".

charludo · 2026-08-24T09:23:40Z

+    fi
+    mkdir -p ./{{ workspace_dir }}
+    echo "Setting up registry..."
+    crane registry serve --address localhost:5000 > ./{{ workspace_dir }}/registry.log 2>&1 &


Lots of places with the 5000 now, how about policy_registry := "localhost:5000" at the top of the file?

charludo · 2026-08-24T09:24:27Z

+    printf "$busybox=localhost:5000/$busybox\n" >> ./{{ workspace_dir }}/just.containerlookup
+    crane copy ghcr.io/edgelesssys/kubernetes/pause:3.6 localhost:5000/kubernetes/pause:3.6
+    echo "Registry is ready."
+    tail -n 0 --pid $PID -f ./{{ workspace_dir }}/registry.log


I think this won't work on darwin...? I have no way to check though.

Correct, this would need to use gtail via Nix.

charludo · 2026-08-24T09:32:17Z

-    actualModRoot=$modRoot
-    modRoot=.
-    ${contrast.cli.preConfigure}
+    install -D ${kata.genpolicy.settings-dev}/genpolicy-settings.json policy-test/assets/genpolicy-settings-kata.json


Interestingly, this now leaves the CLI's asset at the THIS FILE IS REPLACED... state. It currently does not matter, because you always pass --settings= in policy-test/main.go, but maybe worth a comment here.

davidweisse requested review from burgerdev and charludo August 10, 2026 13:37

davidweisse added the no changelog PRs not listed in the release notes label Aug 10, 2026

davidweisse force-pushed the dav/suite-local-registry branch from fc7a027 to 1f0f237 Compare August 12, 2026 10:54

davidweisse changed the base branch from main to dav/policy-test-suite August 12, 2026 10:55

davidweisse force-pushed the dav/policy-test-suite branch 2 times, most recently from 667f516 to ecd4a47 Compare August 17, 2026 11:31

policy-test: init test runner

7c9e278

davidweisse force-pushed the dav/policy-test-suite branch from ecd4a47 to 7c9e278 Compare August 19, 2026 11:12

policy-test: use local registry

15decf2

davidweisse force-pushed the dav/suite-local-registry branch from 1f0f237 to 15decf2 Compare August 19, 2026 12:17

davidweisse marked this pull request as ready for review August 20, 2026 11:50

charludo reviewed Aug 24, 2026

View reviewed changes

davidweisse force-pushed the dav/policy-test-suite branch from 7c9e278 to e6ce525 Compare August 26, 2026 10:56

Base automatically changed from dav/policy-test-suite to main August 26, 2026 13:01

Uh oh!

Conversation

davidweisse commented Aug 10, 2026

Uh oh!

linear-code Bot commented Aug 10, 2026

Uh oh!

davidweisse commented Aug 19, 2026

Uh oh!

charludo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

charludo left a comment •

edited

Loading