Adding atunnel - networking component within ateom by LiorLieberman · Pull Request #559 · agent-substrate/substrate

Lior Lieberman (LiorLieberman) · 2026-07-28T02:39:26Z

[Will rebase after its lgtm]

Introducing atunnel, a component that sits within ateom and handles all inbound and outbound traffic to the actor. With this change in place there is no direct traffic outside and inside the actor with the exception of DNS and non-TCP which traffic which is still TBD.

this PR;

Introduces atunnel, on ingress, it terminates tls and forward to the actor on port 80 (We'll have separate discussions on supporting multiple ports). On egress, it initiates CONNECT over mtls.
Currently atunnel is using the pod/worker identity for mtls, which is wrong. We should likely use the actor identity once Taahir Ahmed (@ahmedtd) work is done on the SessionIdentity (likely to be renamed to something else) to be able to mint certs and JWTs.
RunRequest/RestoreRequest has actor_version and egress_gateway_address here because ateom and atunnel consume them in the same functions that install the nftables rules. Nothing populates either field in this PR, so they arrive zero-valued, so the egress client is never built and no REDIRECT rule is installed. A fast follow-up egress gateway PR adds the producer side that configures these fields.

Huge part of it uses and is built-upon John Howard (@howardjohn) 's work on #460

More specific notes:

New internal/atunnel package — an mTLS ingress server plus a transparently-intercepted egress TCP proxy. Only the ingress half is live in this PR.
Every worker pod hosts an atunnel ingress server on :443. The server authorizes the caller and checks the Actor is the one currently assigned to that worker.
atenet-router dials the tunnel instead of the Actor. The dynamic_forward_proxy cluster is replaced by an ORIGINAL_DST cluster driven by an ext_proc-set x-ate-original-dst header, with upstream mTLS and SPIFFE URI SAN validation. :authority is left untouched, so atunnel can authorize on the original Host.
The pod-IP:80 → actor-veth:80 DNAT is removed. Worker port 80 is no longer an ingress path; the mTLS listener is the only way in.
The egress proxy, the nftables REDIRECT rule and the two new ateompb fields are all gated on an egress gateway address that nothing populates yet, so Actor egress still uses the existing masquerade, unchanged. A pluggable-egress PR is a fast follow.
Tests pass
Appropriate changes to documentation are included in the PR

haiyanmeng · 2026-07-28T13:12:12Z

FYI, #557 maps HTTPS requests to actor port 443 with TLS re-origination- #557

haiyanmeng · 2026-07-28T14:37:38Z

+    extracts the Actor name and Atespace from the `Host` header and calls the
+    Control Plane to resume the Actor and resolve its current worker assignment.
+
+  * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an


Change this to Worker Ingress Tunnel

haiyanmeng · 2026-07-28T14:37:52Z

+  * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an
+    authenticated TLS tunnel to the worker's `atunnel` listener on port 443.
+    `atunnel`, hosted by `ateom`, forwards the request to the active Actor over
+    its private veth interface. Worker pod port 80 is not a direct Actor ingress


pod -> Pod

haiyanmeng · 2026-07-28T14:43:26Z

    participant Atelet as atelet
-    participant Ateom as ateom
+    participant Tunnel as ateom / atunnel
+    participant Actor


This does not render correctly, because Actor is a reserved keyword in Mermaid diagrams. Renaming it works:

sequenceDiagram actor Client participant DNS as atenet DNS participant Gateway as atenet-router participant API as ate-api-server participant Atelet as atelet participant Tunnel as ateom / atunnel participant A as Actor participant Store as snapshot storage Client->>DNS: resolve actor DNS name DNS-->>Client: ingress gateway address Client->>Gateway: HTTP request (Host = actor) Gateway->>API: ResumeActor(atespace, actor name) API->>Atelet: Restore Store-->>Atelet: download snapshot Atelet->>Tunnel: RestoreWorkload Note over Tunnel: restore sandbox and activate Actor network Tunnel-->>Atelet: ready Atelet-->>API: worker pod IP API-->>Gateway: worker assignment Gateway->>Tunnel: mTLS tunnel to worker port 443 Tunnel->>A: forward request to Actor port A-->>Tunnel: response Tunnel-->>Gateway: response Gateway-->>Client: response Note over API,Store: later: an explicit SuspendActor checkpoints back to storage and frees the worker

need to take a whole look at the diagram before we merge. It will be fixed

haiyanmeng · 2026-07-28T15:10:10Z

+    extracts the Actor name and Atespace from the `Host` header and calls the
+    Control Plane to resume the Actor and resolve its current worker assignment.
+
+  * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an


Worthy adding a one-line like egress redirection through atunnel is implemented but not yet enabled, which would stop the readers from concluding either that it's live or that it doesn't exist.

John Howard (howardjohn) · 2026-07-28T15:23:33Z

+	ActorVersionHeader = "X-Ate-Actor-Version"
+)
+
+// ClientConfig configures an egress CONNECT client.


I think we should probably use CONNECT on both directions.

This makes it symmetrical. Least important reason but we already have a 'sunk cost' on CONNECT.

This more clearly future proofs us to multi-port actors. Because right now we have to send to atunnel port 443 which is then making multiple ports hard. Vs sending to some which then can easily forward to any port on the actor.

We can pass additional metadata more clearly. We already are ~abusing the same HTTP channel to send actor responses OR atunnel responses (StaleAssignmentHeader). CONNECT lets these be clearly split

I agree we should. and I think we will. Was working on getting something that unifies all discussions (see #296 (comment) for ref).

For the purpose of getting something in, I think we should split it but I will try to come up with issue/doc/comment today about solving the multi-port (and DNS with double wildcard).

haiyanmeng · 2026-07-28T16:10:24Z

@@ -53,13 +56,15 @@ func NewRouterCmd() *cobra.Command {
 	cmd.Flags().IntVar(&cfg.StatusPort, "status-port", 4040, "Port to serve /statusz on (set <= 0 to disable serving status)")
 	cmd.Flags().DurationVar(&cfg.HealthInterval, "health-interval", 1*time.Second, "Interval for checking health of dependent services")
 	cmd.Flags().IntVar(&cfg.HttpsPort, "port-https", 8443, "TCP port for HTTPS workload traffic entering through the Envoy Router")
-	cmd.Flags().StringVar(&cfg.EnvoyCertPath, "envoy-cert-path", "", "Path to the Envoy certificate file.")
+	cmd.Flags().StringVar(&cfg.EnvoyCertPath, "envoy-cert-path", "", "Path to the Envoy certificate file (if empty, a self-signed cert will be generated for testing)")
+	cmd.Flags().StringVar(&cfg.UpstreamClientCertPath, "upstream-client-cert-path", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "PEM credential bundle (cert+key) the router presents as the client cert when dialing the actor's atunnel ingress server over mTLS. Empty disables upstream mTLS (legacy plaintext pod-IP:80).")


Can we use the same flag surffix as the --atunnel-credential-bundle flag of ateom-gvisor|microvm? Consistency matters more than exactly which one you pick.

haiyanmeng · 2026-07-28T16:10:44Z

-	cmd.Flags().StringVar(&cfg.EnvoyCertPath, "envoy-cert-path", "", "Path to the Envoy certificate file.")
+	cmd.Flags().StringVar(&cfg.EnvoyCertPath, "envoy-cert-path", "", "Path to the Envoy certificate file (if empty, a self-signed cert will be generated for testing)")
+	cmd.Flags().StringVar(&cfg.UpstreamClientCertPath, "upstream-client-cert-path", "/run/podidentity.podcert.ate.dev/credential-bundle.pem", "PEM credential bundle (cert+key) the router presents as the client cert when dialing the actor's atunnel ingress server over mTLS. Empty disables upstream mTLS (legacy plaintext pod-IP:80).")
+	cmd.Flags().StringVar(&cfg.UpstreamTrustPath, "upstream-trust-bundle-path", "/run/podidentity.podcert.ate.dev/trust-bundle.pem", "PEM trust bundle used to validate the actor's atunnel ingress server certificate.")


Can we use the same flag surffix as the --atunnel-trust-bundle flag of ateom-gvisor|microvm? Consistency matters more than exactly which one you pick.

Lior Lieberman (LiorLieberman) · 2026-07-28T17:20:12Z

Review that before we merge - this is ai generated fully

atunnel is the in-worker component that carries Actor traffic in both directions over authenticated channels. * Server: an mTLS HTTPS listener that terminates connections from the ingress gateway, authorizes the peer by its SPIFFE identity, checks that the request targets the Actor currently activated on this worker, and reverse-proxies to the Actor over the private veth. * Egress: an activation-aware TCP proxy for transparently intercepted Actor connections. It resolves the original destination via SO_ORIGINAL_DST and forwards through an EgressDialer, asserting the Actor identity to the far end. Dormant until an egress gateway is configured; the client and dialer land here so the package is reviewed as one unit. Activate/Deactivate bracket an Actor activation so traffic is only carried while an Actor is assigned to the worker, and Deactivate drains in-flight streams before the Actor network is torn down.

Every worker pod now hosts an atunnel ingress server on :443 and an atunnel egress listener, both long-lived, with per-activation Activate/Deactivate bracketing every Run, Restore and Checkpoint. The nftables rules ateom installs change accordingly: * The pod-IP:80 -> actor-veth:80 DNAT is gone. Worker port 80 is no longer an Actor ingress path; the only way in is the mTLS listener on :443, which authorizes the caller and checks the Actor is the one currently assigned to this worker. * A prerouting REDIRECT sends Actor TCP egress to the local atunnel egress listener, preserving SO_ORIGINAL_DST. It is only installed when the activation carries an egress gateway address, which nothing populates yet, so the masquerade path is unchanged and Actor egress behaves exactly as before. The ateom Run/Restore protos gain the two fields that arm that path: egress_gateway_address, which decides whether the redirect is installed at all, and actor_version, the Actor resource version ate-api observed when assigning the worker, which atunnel asserts to the egress gateway as a lower bound on trustworthy Actor metadata. Both are consumed here and left unset. atelet and ate-api start populating them in the egress gateway change, which is the point at which they mean anything, so this change adds no new requirement to the atelet wire contract. atecontroller gives worker pods the podidentity credential + trust bundles (the atunnel server identity), the servicedns trust bundle, and container port 443. podidentitysigner now issues certs with ExtKeyUsageServerAuth as well as ClientAuth, without which the worker cannot present its podidentity cert as a TLS server cert and the gateway handshake fails.

The router used to rewrite :authority to the actor's worker pod IP and let the dynamic_forward_proxy cluster resolve it, reaching the actor over plaintext pod-IP:80. The worker no longer DNATs pod-IP:80 to the actor, so that path is gone. Instead the ext_proc resolves the actor to its worker's atunnel ingress address (IP:443) and puts it in x-ate-original-dst. An ORIGINAL_DST cluster dials exactly that address, which leaves the request Host as the actor DNS name -- atunnel needs it to authorize the active actor. The header is set with OVERWRITE_IF_EXISTS_OR_ADD so a client-supplied value can never influence the address Envoy dials. The upstream hop is mTLS: the cluster presents the router's podidentity credential bundle as its client cert and validates the atunnel server against the podidentity trust bundle. Validation matches the SPIFFE URI SAN prefix rather than the dialed pod IP, because the atunnel cert carries only a spiffe:// URI SAN and Envoy's default SAN check against an ephemeral pod IP would never match. atunnel in turn only accepts spiffe://cluster.local/ns/ate-system/sa/atenet-router. The dynamic_forward_proxy cluster, HTTP filter and DNS cache config are removed along with the :authority rewrite.

IdentityFile /.ssh/redshell_private_key Bring the architecture and threat-model docs in line with the atunnel ingress path: the router no longer rewrites :authority to a worker pod IP and forwards over plaintext port 80, it opens an mTLS tunnel to atunnel on worker port 443, which forwards to the Actor over its private veth. Drop cmd/atenet/atenet-diagram.png. It predates the ext_proc/ORIGINAL_DST design and is now wrong in the part that matters most. The README section it illustrated gains a short accurate note about the upstream hop instead of a dangling image reference. Add an e2e suite covering the change end to end: TestActorDirectAccess asserts that the worker pod's port 80 is no longer a reachable Actor ingress path (the DNAT rule is gone) and that the same Actor still answers /readyz through atenet-router over the atunnel mTLS hop. It uses the counter demo as its fixture, so it only needs --deploy-demo-counter. internal/e2e/testmain.go picks up ParseSkippedFlags so that `go test` flags (-run, -v, ...) survive pflag parsing and reach the suite.

Bowei Du (bowei) · 2026-07-29T18:44:09Z

+	UpstreamTrustPath      string
+	// UpstreamSpiffePrefix validates the actor's atunnel server cert by its
+	// SPIFFE URI SAN prefix (trust domain) instead of the dialed pod IP.
+	UpstreamSpiffePrefix string


nit: put a new line to separate this in its own block

Lior Lieberman (LiorLieberman) · 2026-07-29T19:43:05Z

+              signerName: podidentity.podcert.ate.dev/identity
+              labelSelector:
+                matchLabels:
+                  podcert.ate.dev/canarying: live


TODO: not sure if we need this or not. Discuss with other community members

Lior Lieberman (LiorLieberman) requested a review from Bowei Du (bowei) July 28, 2026 02:42

haiyanmeng mentioned this pull request Jul 28, 2026

atenet: map HTTPS requests to actor port 443 with TLS re-origination #557

Open

haiyanmeng reviewed Jul 28, 2026

View reviewed changes

John Howard (howardjohn) reviewed Jul 28, 2026

View reviewed changes

haiyanmeng reviewed Jul 28, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) commented Jul 28, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) force-pushed the pr1-atunnel-ingress branch from e46231f to ac569ad Compare July 28, 2026 20:32

Lior Lieberman (LiorLieberman) added 3 commits July 28, 2026 13:32

Lior Lieberman (LiorLieberman) force-pushed the pr1-atunnel-ingress branch from ac569ad to a0c14f2 Compare July 29, 2026 00:02

Lior Lieberman (LiorLieberman) force-pushed the pr1-atunnel-ingress branch from a0c14f2 to 6030db8 Compare July 29, 2026 01:28

Bowei Du (bowei) reviewed Jul 29, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) commented Jul 29, 2026

View reviewed changes

Uh oh!

Conversation

Lior Lieberman (LiorLieberman) commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haiyanmeng commented Jul 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Lior Lieberman (LiorLieberman) commented Jul 28, 2026 •

edited

Loading