fix(policy): propose rules for unknown DNS hosts (#3707)

* fix(policy): propose rules for unknown DNS hosts

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* docs(policy): clarify synthetic DNS use across protocols

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(policy): harden unknown-host DNS observations

- Emit the policy_dns_ineligible denial for every unknown name and
  report observation staging failures as DNS failure events.
- Refuse unknown names during fail-closed quarantine and after the
  observation budget, now a quarter of each address family's pool.
- Pin transparent TCP to the mapping of the deciding policy generation
  so a reload between DNS and authorization fails closed.
- Stop Docker workloads from inheriting host DNS search domains, which
  let the first expanded short name claim an observation address.
- Share mechanistic draft polling in conformance, register
  new-hostname-proposal in the installed suite, and update docs.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* test(policy): build policy DNS proxy tests on every target

The proxy tests name PolicyEndpointId, which proxy.rs imported only on
Linux, so the macOS test build failed. Import it for test builds too.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(policy): name DNS queries and mapped hosts in OCSF denials

DNS denial and failure events attached port 53 to the queried name,
which read as a connection to that host. They now carry only the name.
Transparent TCP denials for a policy DNS address show the mapped
hostname and keep the synthetic address in dst_endpoint.ip.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Author: pimlock

CI.md

@@ -50,8 +50,8 @@ Manually dispatch `Integration Tests` on the candidate branch with an
 [{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]
 ```
 
-This runs the `mechanistic-proposal` and `policy-local` conformance tests in the
-installed-artifact suite. The artifact run must contain the candidate CLI and
+This runs the `mechanistic-proposal`, `new-hostname-proposal`, and
+`policy-local` conformance tests in the installed-artifact suite. The artifact run must contain the candidate CLI and
 gateway binaries and runtime images. This manual run does not replace the
 required PR E2E gate.
 

architecture/sandbox.md

@@ -269,8 +269,9 @@ Input mediation, DNS/TCP authorization, and outer-fence enforcement are
 identical in both modes. The selected mode is emitted in the sandbox
 qualification output (`seccomp_listener_mode`).
 
-DNS uses an exact sandbox-local resolver at `127.0.0.53:53`. The driver sets the
-nameserver and permits an unprivileged bind to port 53. UDP and TCP DNS requests
+DNS uses an exact sandbox-local resolver at `127.0.0.53:53`. The driver sets that
+nameserver without search domains, so names reach policy DNS as the workload
+wrote them, and permits an unprivileged bind to port 53. UDP and TCP DNS requests
 are forwarded through the supervisor, which applies hostname-based DNS policy.
 The Podman driver supplies that resolver configuration as a driver-owned,
 read-only secret mounted at `/etc/resolv.conf`; the workload remains on
@@ -323,10 +324,12 @@ preserves this lifetime rule; persistent responses remain eligible for reuse.
 An explicit `protocol: tcp` endpoint with a valid DNS hostname opts into native
 DNS and transparent TCP when the selected runtime advertises that substrate.
 Hostless `allowed_ips` and literal-IP selectors remain available only to the
-legacy explicit-proxy path when `protocol` is omitted. The shared supervisor
-answers only eligible DNS names, returns an epoch-scoped synthetic address, and
+legacy explicit-proxy path when `protocol` is omitted. For an eligible DNS
+name, the shared supervisor returns an epoch-scoped synthetic address and
 publishes the expiring name, endpoint, ports, policy generation, and validated
-real addresses as one correlation. A connection to that synthetic address is
+real addresses as one correlation. A name absent from policy receives at most a
+contract-free observation address for policy advisor proposals; see
+[Security Policy](security-policy.md). A connection to that synthetic address is
 captured before the bypass fence, mapped back to its workload process, authorized
 through the same egress pipeline, and dialed only through the pinned addresses.
 Omitted protocol endpoints retain explicit-proxy behavior.

architecture/security-policy.md

@@ -282,6 +282,19 @@ because it changes the effective access model for every sandbox on the gateway.
 The policy advisor pipeline turns observed denials into draft policy
 recommendations. There are two proposers (sandbox-side mechanistic mapper,
 agent-authored via `policy.local`); the gateway is the single referee.
+For a new DNS name absent from policy, the supervisor can
+publish a bounded, short-lived synthetic observation mapping without querying
+an upstream resolver. It carries only the name to the subsequent TCP mediation
+step, which supplies the destination port and verified process identity for a
+denial summary. Observation mappings have no endpoint contracts or real pinned
+addresses, cannot authorize a relay, and become stale on policy generation
+change. Transparent TCP re-acquires the mapping at the generation that
+authorized the connection, so a reload between DNS and authorization fails
+closed instead of resolving the name again. Every unknown name still produces a
+DNS denial event. Reserved names, fail-closed quarantine, and an exhausted
+observation budget (a quarter of each address family's synthetic pool) keep the
+plain DNS refusal. This mechanistic observation path does not depend on the
+agent-authored proposal setting.
 When enabled, L7 `policy_denied` responses include both structured
 `next_steps` and a short `agent_guidance` string so generic agents can continue
 through the proposal loop instead of treating the denial as terminal.

crates/openshell-conformance/src/lib.rs

@@ -24,8 +24,8 @@ use tokio::time::sleep;
 use self::executor::{CliExecutionError, CliExecutor, ProcessCli};
 
 pub use scenarios::{
-    MECHANISTIC_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO, SANDBOX_LIFECYCLE_SCENARIO,
-    SMOKE_SCENARIO,
+    MECHANISTIC_PROPOSAL_SCENARIO, NEW_HOSTNAME_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO,
+    SANDBOX_LIFECYCLE_SCENARIO, SMOKE_SCENARIO,
 };
 
 /// An installed conformance scenario.
@@ -48,6 +48,7 @@ const SCENARIOS: &[Scenario] = &[
     SMOKE_SCENARIO,
     SANDBOX_LIFECYCLE_SCENARIO,
     MECHANISTIC_PROPOSAL_SCENARIO,
+    NEW_HOSTNAME_PROPOSAL_SCENARIO,
     POLICY_LOCAL_SCENARIO,
 ];
 

crates/openshell-conformance/src/scenarios/mod.rs

@@ -7,6 +7,8 @@ mod policy_behavior;
 mod sandbox_lifecycle;
 mod smoke;
 
-pub use policy_behavior::{MECHANISTIC_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO};
+pub use policy_behavior::{
+    MECHANISTIC_PROPOSAL_SCENARIO, NEW_HOSTNAME_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO,
+};
 pub use sandbox_lifecycle::SANDBOX_LIFECYCLE_SCENARIO;
 pub use smoke::SMOKE_SCENARIO;

crates/openshell-conformance/src/scenarios/policy_behavior.rs

@@ -37,6 +37,21 @@ pub const MECHANISTIC_PROPOSAL_SCENARIO: Scenario = Scenario {
     run: run_mechanistic_proposal,
 };
 
+pub const NEW_HOSTNAME_PROPOSAL_SCENARIO: Scenario = Scenario {
+    name: "new-hostname-proposal",
+    description: "Turn a denied TCP open to a hostname absent from policy into a scoped draft.",
+    run: run_new_hostname_proposal,
+};
+
+const EMPTY_NETWORK_POLICY: &[u8] = br"version: 1
+filesystem_policy:
+  include_workdir: true
+  read_only: [/usr, /bin, /lib, /lib64, /proc, /dev/urandom, /app, /etc, /var/log]
+  read_write: [/sandbox, /tmp, /dev/null]
+landlock: { compatibility: best_effort }
+network_policies: {}
+";
+
 fn run_policy_local(runner: &mut OpenShellRunner) -> ScenarioFuture<'_> {
     Box::pin(async move {
         let name = format!("ct-{}-pl", runner.id());
@@ -279,16 +294,7 @@ fn run_mechanistic_proposal(runner: &mut OpenShellRunner) -> ScenarioFuture<'_>
     Box::pin(async move {
         let mut policy = NamedTempFile::new().map_err(|error| error.to_string())?;
         policy
-            .write_all(
-                br"version: 1
-filesystem_policy:
-  include_workdir: true
-  read_only: [/usr, /bin, /lib, /lib64, /proc, /dev/urandom, /app, /etc, /var/log]
-  read_write: [/sandbox, /tmp, /dev/null]
-landlock: { compatibility: best_effort }
-network_policies: {}
-",
-            )
+            .write_all(EMPTY_NETWORK_POLICY)
             .map_err(|error| error.to_string())?;
         let policy_path = policy
             .path()
@@ -345,60 +351,139 @@ network_policies: {}
             return Err(probe.failure_diagnostic("TCP open is denied before any upstream dial"));
         }
 
-        let started = Instant::now();
-        loop {
-            let draft = runner
-                .step("mechanistic-draft")
-                .description("a single scoped mechanistic draft appears")
-                .with_timeout(COMMAND_TIMEOUT)
-                .run(&["rule", "get", &name])
-                .await
-                .map_err(|error| error.to_string())?;
-            if !draft.success() {
-                if started.elapsed() >= PROPOSAL_TIMEOUT {
-                    return Err(draft.failure_diagnostic("the reviewer inbox is readable"));
-                }
-                sleep(POLL_INTERVAL).await;
-                continue;
+        await_mechanistic_draft(
+            runner,
+            &name,
+            &ExpectedDraft {
+                rule: "allow_1_1_1_1_443",
+                endpoint: "1.1.1.1:443",
+                binary: &binary,
+            },
+            probe.stderr(),
+        )
+        .await
+    })
+}
+
+fn run_new_hostname_proposal(runner: &mut OpenShellRunner) -> ScenarioFuture<'_> {
+    Box::pin(async move {
+        let mut policy = NamedTempFile::new().map_err(|error| error.to_string())?;
+        policy
+            .write_all(EMPTY_NETWORK_POLICY)
+            .map_err(|error| error.to_string())?;
+        let policy_path = policy
+            .path()
+            .to_str()
+            .ok_or("temporary policy path is not UTF-8")?;
+        let name = format!("ct-{}-nh", runner.id());
+        create_sandbox(runner, &name, Some(policy_path)).await?;
+        let binary = sandbox_bash_path(runner, &name).await?;
+
+        let probe = runner
+            .step("denied-new-hostname")
+            .description("Bash cannot connect to pypi.org:80 before approval")
+            .with_timeout(COMMAND_TIMEOUT)
+            .run(&[
+                "sandbox",
+                "exec",
+                "--name",
+                &name,
+                "--no-tty",
+                "--",
+                "bash",
+                "-c",
+                "if exec 3<>/dev/tcp/pypi.org/80; then echo UNEXPECTED_ALLOWED; exit 1; else echo DENIED; fi",
+            ])
+            .await
+            .map_err(|error| error.to_string())?;
+        probe.require_success()?;
+        if !probe.stdout().lines().any(|line| line == "DENIED") {
+            return Err(probe.failure_diagnostic("new hostname stays denied before approval"));
+        }
+
+        await_mechanistic_draft(
+            runner,
+            &name,
+            &ExpectedDraft {
+                rule: "allow_pypi_org_80",
+                endpoint: "pypi.org:80",
+                binary: &binary,
+            },
+            probe.stderr(),
+        )
+        .await
+    })
+}
+
+/// The single L4 mechanistic draft expected for one denied endpoint.
+struct ExpectedDraft<'a> {
+    rule: &'a str,
+    endpoint: &'a str,
+    binary: &'a str,
+}
+
+/// Poll the reviewer inbox until a draft appears, tolerating transient read
+/// failures, then require that it is the single expected draft.
+async fn await_mechanistic_draft(
+    runner: &OpenShellRunner,
+    sandbox: &str,
+    expected: &ExpectedDraft<'_>,
+    probe_stderr: &str,
+) -> Result<(), String> {
+    let started = Instant::now();
+    loop {
+        let draft = runner
+            .step("mechanistic-draft")
+            .description("a single scoped mechanistic draft appears")
+            .with_timeout(COMMAND_TIMEOUT)
+            .run(&["rule", "get", sandbox])
+            .await
+            .map_err(|error| error.to_string())?;
+        if !draft.success() {
+            if started.elapsed() >= PROPOSAL_TIMEOUT {
+                return Err(draft.failure_diagnostic("the reviewer inbox is readable"));
             }
-            if !draft.stdout().contains("Chunk:") {
-                if started.elapsed() >= PROPOSAL_TIMEOUT {
-                    return Err(draft.failure_diagnostic(&format!(
-                        "one mechanistic draft for 1.1.1.1:443 and {binary}; probe stderr:\n{}",
-                        probe.stderr()
-                    )));
-                }
-                sleep(POLL_INTERVAL).await;
-                continue;
+            sleep(POLL_INTERVAL).await;
+            continue;
+        }
+        if !draft.stdout().contains("Chunk:") {
+            if started.elapsed() >= PROPOSAL_TIMEOUT {
+                return Err(draft.failure_diagnostic(&format!(
+                    "one mechanistic draft for {} and {}; probe stderr:\n{probe_stderr}",
+                    expected.endpoint, expected.binary
+                )));
             }
-            assert_mechanistic_draft(draft.stdout(), &binary)
-                .map_err(|error| draft.failure_diagnostic(&error))?;
-            return Ok(());
+            sleep(POLL_INTERVAL).await;
+            continue;
         }
-    })
+        return assert_mechanistic_draft(draft.stdout(), expected)
+            .map_err(|error| draft.failure_diagnostic(&error));
+    }
 }
 
-fn assert_mechanistic_draft(output: &str, binary: &str) -> Result<(), String> {
+fn assert_mechanistic_draft(output: &str, expected: &ExpectedDraft<'_>) -> Result<(), String> {
     let fields = output.lines().map(str::trim).collect::<Vec<_>>();
     let field = |name: &str| {
         fields
             .iter()
             .find_map(|line| line.strip_prefix(name).map(str::trim))
     };
+    let endpoints = format!("{} [L4]", expected.endpoint);
     if fields
         .iter()
         .filter(|line| line.starts_with("Chunk:"))
         .count()
         != 1
         || !matches!(field("Status:"), Some("pending" | "approved"))
-        || field("Rule:") != Some("allow_1_1_1_1_443")
-        || field("Binary:") != Some(binary)
-        || field("Binaries:") != Some(binary)
-        || field("Endpoints:") != Some("1.1.1.1:443 [L4]")
-        || !field("Rationale:").is_some_and(|value| value.contains("1.1.1.1:443"))
+        || field("Rule:") != Some(expected.rule)
+        || field("Binary:") != Some(expected.binary)
+        || field("Binaries:") != Some(expected.binary)
+        || field("Endpoints:") != Some(endpoints.as_str())
+        || !field("Rationale:").is_some_and(|value| value.contains(expected.endpoint))
     {
         return Err(format!(
-            "expected one pending or approved L4 mechanistic draft scoped to {binary} and 1.1.1.1:443"
+            "expected one pending or approved L4 mechanistic draft scoped to {} and {}",
+            expected.binary, expected.endpoint
         ));
     }
     Ok(())
@@ -462,11 +547,36 @@ async fn create_sandbox(
 
 #[cfg(test)]
 mod tests {
-    use super::assert_mechanistic_draft;
+    use super::{ExpectedDraft, assert_mechanistic_draft};
+
+    const EXPECTED: ExpectedDraft<'static> = ExpectedDraft {
+        rule: "allow_pypi_org_80",
+        endpoint: "pypi.org:80",
+        binary: "/usr/bin/bash",
+    };
+
+    fn draft(binary: &str) -> String {
+        format!(
+            "Chunk: id\nStatus: pending\nRule: allow_pypi_org_80\nBinary: {binary}\nRationale: Allow {binary} to connect to pypi.org:80 (HTTP).\nEndpoints: pypi.org:80 [L4]\nBinaries: {binary}\n"
+        )
+    }
+
+    #[test]
+    fn draft_assertion_accepts_a_hostname_scoped_draft() {
+        assert!(assert_mechanistic_draft(&draft("/usr/bin/bash"), &EXPECTED).is_ok());
+    }
 
     #[test]
     fn draft_assertion_rejects_unrelated_binary() {
-        let draft = "Chunk: id\nStatus: pending\nRule: allow_1_1_1_1_443\nBinary: /usr/bin/sh\nRationale: Allow sh to connect to 1.1.1.1:443.\nEndpoints: 1.1.1.1:443 [L4]\nBinaries: /usr/bin/sh\n";
-        assert!(assert_mechanistic_draft(draft, "/usr/bin/bash").is_err());
+        assert!(assert_mechanistic_draft(&draft("/usr/bin/sh"), &EXPECTED).is_err());
+    }
+
+    #[test]
+    fn draft_assertion_rejects_fields_spread_across_drafts() {
+        let drafts = format!(
+            "{}Chunk: other\nStatus: pending\nRule: allow_1_1_1_1_443\n",
+            draft("/usr/bin/bash")
+        );
+        assert!(assert_mechanistic_draft(&drafts, &EXPECTED).is_err());
     }
 }

crates/openshell-driver-docker/src/lib.rs

@@ -5796,6 +5796,10 @@ fn build_container_create_body_for_image(
             }),
             network_mode: Some("none".to_string()),
             dns: Some(vec!["127.0.0.53".to_string()]),
+            // Docker otherwise copies the host's search domains. Like the
+            // Podman, Kubernetes, and VM resolvers, send names to policy DNS
+            // exactly as written so short names never expand to host domains.
+            dns_search: Some(vec![".".to_string()]),
             tmpfs: Some(HashMap::from([(
                 openshell_sandbox_backend::SUPERVISOR_CA_RUNTIME_DIR.to_string(),
                 format!(

crates/openshell-driver-docker/src/tests.rs

@@ -1278,6 +1278,7 @@ fn container_creation_uses_inspected_immutable_image() {
     );
     assert_eq!(host.network_mode.as_deref(), Some("none"));
     assert_eq!(host.dns, Some(vec!["127.0.0.53".to_string()]));
+    assert_eq!(host.dns_search, Some(vec![".".to_string()]));
 }
 
 #[test]
@@ -2714,6 +2715,11 @@ fn build_container_create_body_disables_docker_networking() {
     );
     assert_eq!(host_config.extra_hosts, None);
     assert_eq!(host_config.dns, Some(vec!["127.0.0.53".to_string()]));
+    assert_eq!(
+        host_config.dns_search,
+        Some(vec![".".to_string()]),
+        "host search domains must not expand workload names before policy DNS"
+    );
 }
 
 #[test]