rfc: client-side proxy

[burgerdev]

rendered version

[charludo]

Some preliminary thoughts

[sespiros]

Suggest pinning a manifest here instead of a CA cert, with the proxy verifying via the SDK and deriving the mesh CA itself: a static CA file goes stale on every manifest update, and the manifest is the actual trust anchor. privatemode-proxy already does exactly this (GetAttestedMeshCA, with static and dynamic manifest modes), so this could reuse that machinery instead of introducing a weaker parallel path.

[burgerdev]

If anything, we'd need to pin a manifest history, not only a manifest. PM gets away with this because the deployment is immutable (iirc). The pinned history would be equally stale on manifest updates, though, and cause more connectivity headache (Coordinator needs to be exposed when the proxy comes up).

[sespiros]

On the PM point: yes, its attest getter hard-fails unless the history has exactly one manifest (which we can't assume for a generic Contrast deployment). So history is actually the object to pin.

On the staleness point: a stale CA fails with a more generic TLS error while a stale manifest pin fails as "Coordinator enforces manifest X, expected Y". Also it's a question of what you distribute, a CA (which is a derived artifact) or a new manifest. The manifest (or a manifest history) could also be distributed via a remote source that we publish similar to what the pm proxy does when you don't pass a manifest to it.

On the connectivity point: the CA has the same dependency no? Coordinator needs to be exposed also to whoever runs verify (in order to then distribute the CA to the proxy). If a reverse proxy exists, it could relay to the Coordinator too.

[burgerdev]

I added a section for why I don't mean to solve this here.

[burgerdev]

Folks, I updated the proposal, ptal.

[sespiros]

Thanks for the changes.

[sespiros]

One more setup prerequisite for this example: web.cc.cloud must be added to the web service's manifest SANs, so its mesh certificate is valid for that name. That's the manual manifest-edit flow the deployment guide documents for load balancer IPs. generate only emits the resource name. Note the default * SAN doesn't cover this despite what the manifest docs suggest: a bare wildcard fails RFC 6125-style matching in Go, OpenSSL, and curl (tracked separately in CON-222).

[burgerdev]

Yes, that's indeed missing.

[sespiros]

Suggested change

	4. The proxy receives the `ClientHello` and initiates a similar TLS connection (ALPN) to the backend, verified with the Contrast CA certificate.
	4. The proxy receives the `ClientHello` and initiates a similar TLS connection (ALPN) to the backend, verifying the backend's certificate chain against the configured Contrast CA certificate and its SANs against the `CONNECT` target (`web.cc.cloud`), which the backend's mesh certificate carries per the setup above.

[sespiros]

An attack scenario here: the cluster owner, who is untrusted, controls the infrastructure behind the domain's IP, so they can pass HTTP-01 or TLS-ALPN-01 challenge and obtain a browser-valid certificate for web.cc.cloud from a public CA. Clients that bypass the proxy (i.e off-network devices) and hit the backend directly would then see a green padlock on an impersonation of the confidential service, instead of the hard cert error they'd get today.

Remediation: the org that owns web.cc.cloud controls its DNS (outside the cluster owner's reach and publishes:

web.cc.cloud.  CAA  0 issue "letsencrypt.org; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/12345678; validationmethods=dns-01"

Every CA except Let's Encrypt must then refuse the name. CAs are required to honor CAA by the CA/Browser Forum Baseline Requirements to stay in browser root programs. Let's Encrypt in turn rejects HTTP-01/TLS-ALPN-01 and accepts only DNS-01, whose challenge record lives in the org's DNS zone, which the cluster owner can't write to. The accounturi parameter additionally restricts issuance to the org's own ACME account.

[burgerdev]

The scenario applies when the client does not set the proxy, somehow misconfigures it, or the browser ignores it, right? That's bad, and I don't think the workaround is sufficient, at least not without additional protections on the DNS level (to make sure the CAA record is actually delivered to letsencrypt).

[sespiros]

Yes, and it really depends on the exact setup but I thought to mention it in any case. i.e I was thinking a scenario during which the proxy would not be local but accessible via VPN and a device would access the service while the VPN is off. We should at least cover it in documentation.

Agreed on the second part too although in this model we are already putting trust on some corp-managed network so as a hardening step maybe requiring DNSSEC on top is sufficient. For defending against BGP hijacking there is also MPIC. I mean even then nothing stops a CA which simply violates the requirements. That could be also covered in documentation, that backends shouldn't be directly reachable by clients in the first place.

[sespiros]

nit: browsers don't read the HTTP_PROXY environment variable. They take proxy settings from the OS, a PAC file, or an extension like FoxyProxy. HTTP_PROXY/HTTPS_PROXY is the CLI/library convention (curl, Go, Python), and there it's HTTPS_PROXY that governs https:// URLs.

Suggested change

	The client browser is configured to use `proxy.internal` as its `HTTP_PROXY`.
	The client browser is configured to use `proxy.internal` as its HTTPS proxy`.

[burgerdev]

Wow, this keeps being full of surprises. As far as I can tell, FoxyProxy interprets the HTTPS vs HTTP prefix as the protocol to initiate with the proxy, not the backend - contrary to the interpretation of curl and go.

[sespiros]

hmm right "HTTPS proxy" is the wrong term again, it should be "configured to send traffic through proxy.internal (a standard HTTP CONNECT proxy)"

[sespiros]

nit: both halves of this are stale now. SOCKS5 is broadly usable (Firefox and Chrome natively, curl via socks5h://, ALL_PROXY for most tools), and it wouldn't need a listener per remote service since the SOCKS request carries the target host:port in-band. The actual reason it loses to the reworked design is different: a SOCKS tunnel is transparent, so the browser would run TLS itself and still hit the mesh cert, whereas the re-encrypting CONNECT proxy can satisfy the browser with a web PKI cert. Suggest reframing the bullet around that.

[burgerdev]

Maybe SOCKS is the way to go, then? The idea of this proposal, after the first rework, is that the browser keeps doing TLS, but the proxy MITMs it. We can do the same in SOCKS, no?

[sespiros]

ah a SOCKS proxy that terminates at the proxy, right, I keep thinking about relaying proxies. So yes this is equally viable but I guess we don't need any non-HTTP TCP? My understanding is that the HTTP CONNECT was picked for wider client-config support.

[sespiros]

Suggested change

	This proposal relies on the HTTP `CONNECT` verb and the associated `HTTP_PROXY` settings, which are [widely supported across browsers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods/CONNECT#browser_compatibility) and libraries.
	This proposal relies on the HTTP `CONNECT` tunnel that browsers open for `https://` URLs, which is [widely supported across browsers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods/CONNECT#browser_compatibility) and libraries.

[pskopnik]

I had expected "reverse proxy" in this list (there is a "reverse" mention below, but it seems to refer to the forwarding proxy). Did you disqualify it?

Its architecture/topology would be close to the forwarding proxy proposal and it may be a more familiar deployment pattern for operators. I see some advantages in the changes:

1. Domain name and destination ownership is the same: The web PKI cert & key for app.comp.intra is placed only at the reverse proxy. No need for involvement of the company's interception CA, copying cert & key to two places or double issuing.
2. Decouples the user-facing endpoint completely from the remote cluster endpoint. Responsibilities between both operators are clearly split. Users would not accidentally access the aTLS service directly.
3. The in-cluster gateway only needs to route on L4, not L7.

But the biggest advantage I see is in implementation. There are plenty of mature reverse proxy implementation which one could plug a contrast-atls backend into. Caddy could be interesting because it is Go and has an extension point for upstream transports. General features provided by Caddy would remain available to operators: automatic Let's Encrypt, routing to several backends based on path or domain, auth in front, setting additional headers such as CORS.

[burgerdev]

Sorry for the confusion: the "forward proxy" section is a leftover from the previous design (where it didn't fit, either). This RFC just proposes a regular HTTP proxy that's visible to the clients, while forward and reverse proxies wouldn't be. I'm sticking to the nomenclature from https://www.cloudflare.com/learning/cdn/glossary/reverse-proxy/ here. According to that, a reverse proxy would be running in the untrusted datacenter, so it would need to run in a CVM and be attested (defeating the point).

But I believe what you are suggesting is an on-prem forward proxy? That would neatly solve the problem from https://github.com/edgelesssys/contrast/pull/2509/changes#r3598849781. The downside I see is that such a proxy would require configuration for mapping downstream to upstream URLs. As a local admin, if I need to do that configuration anyway, I could also just add the certificate to Caddy and not require any Edgeless software. Which is a valid alternative, I'd say.

[pskopnik]

Existing terminology does not map cleanly to confidential computing paradigms. (I'd stand by my interpretation of reverse proxy as the authoritatively-determined destination for traffic to a specific backend).

Maybe "on-prem gateway" is better. It would be a contrast-general version of the Privatemode proxy. Topology-wise, that is identical to your set up diagram.

As a local admin, if I need to do that configuration anyway, I could also just add the certificate to Caddy and not require any Edgeless software.

I still lack understanding of the nuances of contrast.
The "add the certificate" variant you describe would be: An admin performs remote attestation against the deployment and receives the mesh CA cert. By adding this to Caddy (or nginx, ...) with the built-in TLS upstream transport, any successful upstream connection implies the upstream endpoint is part of the attested manifest.
The contrast-atls upstream transport variant would be: The admin configures the gateway with a manifest history and the gateway performs the remote attestation inside the gateway eagerly on start / lazily on connection.

Indeed it seems there is no advantage to the contrast-atls transport, if no rotation of the mesh CA is expected. A fresh mesh CA under the same manifest is unlikely for an HA deployment, and a new manifest should not be trusted automatically. So the real advantage would come if there were a protocol to receive or verify a new manifest automatically.

[charludo]

IMO, this version of the proposal makes a lot of sense and I'd be OK with going with it. The one thing I really don't like is the forward proxy a) having access to the plaintext client requests and server responses, and as a consequence b) the proxy being able to unilaterally stop forwarding to the Contrast deployment, and instead forwarding to an attacker-controlled server, without the client having any chance of noticing. The reason for both is that the forward proxy is holding the web PKI secret.

I had an idea for a slightly different architecture; I hope the following makes sense, and does not derail the conversation too much ^^

• let the forward proxy ONLY handle TLS with the Contrast CA certificate
• instead of terminating the web PKI TLS for web.cc.cloud, it receives the so-encrypted request, then wraps it in aTLS with the Contrast CA cert.
• Inside the cluster, an off-the-shelve reverse proxy (nginx, caddy,...) handles ACME for *.cc.cloud (as well the the usual reverse proxy duties)

graph TB
    subgraph OnPrem["On-Premises Environment"]
        Client[Client]
        Bot[Bot]
        Proxy[Proxy]
        Client -->|CONNECT web.cc.cloud:443| Proxy
        Proxy -->|Additionally encrypt| Proxy
        Bot --> Proxy
    end

    subgraph Cloud["Cloud Environment"]
        LB[Gateway]
        subgraph K8s["Kubernetes Cluster"]
            RP[Reverse Proxy]
            App1[Web service]
            App2[Voting service]
        end
            LB --> RP
            RP -->|ACME for *.cc.cloud| RP
            RP --->|web.cc.cloud| App1
            RP --->|voting.cc.cloud| App2
    end

    Proxy ----->|"atls ( pki_tls ( request ) )"| LB

Loading

The major advantage here is that the forward proxy can not fake forwarding the traffic to the CC cluster, because it does not possess the necessary keys to terminate the web PKI TLS traffic.

Additionally, I think we should then easily be able to move the forward proxy from "in-network" to "on-client".

Regardless, the proxied connection then looks like this:

0. The client browses to https://web.cc.cloud.
1. The browser issues a CONNECT web.cc.cloud:443 request to the proxy.
2. The proxy establishes and verifies the aTLS session.
   If that succeeds, the proxy returns 200 OK to the client to indicate tunnel establishment, otherwise a 503 Service Unavailable.
3. The browser initiates a TLS connection over the tunnel, expecting a web PKI certificate for web.cc.cloud.
4. The forward proxy receives the encrypted message from the client. It wraps this message in the aTLS session. It forwards the now doubly encrypted message.
5. The mesh ingress terminates the outer aTLS.
6. The in-cluster reverse proxy terminates the web PKI TLS connection and forwards the request to the service.
7. The forward proxy decryptes the outer aTLS layer and forwards teh still-web-PKI-encrypted response to the client, unable to decrypt the message contents.
8. The forward proxy continues to route traffic back and forth until the sessions close.

In short, the forward proxy becomes "key-less" (and sees no plaintext), this enables it to move to the client, and that in turn strengthens the e2e security story.

In short, the forward proxy becomes key-less, does not handle web-PKI TLS termination and thus sees no plaintext. This allows it to move to the client, and that in turn strengthens the e2e security story. The web-PKI key material lives only inside the CC cluster.