drv-daemon: ad-hoc Builds + queue-runner handoff by amaanq · Pull Request #1692 · NixOS/hydra

amaanq · 2026-04-30T19:31:02Z

Motivation

Hydra currently builds import-from-derivation work on the evaluator host. Those builds bypass the queue runner, builder fleet, Hydra build history, and the normal logs/operators use to debug failures.

Solution

This PR adds hydra-drv-daemon, a Nix daemon protocol service that turns evaluator build requests into ad-hoc Hydra builds. When configured, hydra-evaluator points IFD-capable child processes at this daemon via NIX_REMOTE, so IFDs are built by the normal queue-runner / builder path instead of locally on the evaluator.

The daemon inserts an internal adhoc/adhoc build row, notifies the queue runner, waits for build_finished, and returns the completed build result to the Nix client. Read-side store operations and uploads are proxied to the upstream nix-daemon.

Ericson2314 · 2026-04-30T22:02:30Z

This should not be called this, dynamic derivations and import-from-derivation are completely separate features.

We want one test for "ad hoc jobs" and one test for a regular job (like with the project/jobset stuff) but using import from derivation.

Also this test isn't using content-addressing (on purpose, because IFD is not experimental)

Ericson2314 · 2026-04-30T22:03:06Z

+my $ctx = test_context(
+    nix_config => qq|
+    experimental-features = ca-derivations dynamic-derivations
+    |,
+);
+


Suggested change

my $ctx = test_context(

nix_config => qq|

experimental-features = ca-derivations dynamic-derivations

|,

);

Ericson2314 · 2026-04-30T22:05:14Z

+sub _spawn_builder {
+    my ($self) = @_;
+    my $ctx = $self->{ctx};
+
+    $self->_spawn(
+        builder => "Builder",
+        [
+            "hydra-builder",
+            "--gateway-endpoint", "http://[::1]:$self->{grpc_port}",
+        ],
+        env => {
+            NIX_REMOTE    => $ctx->{builder}{nix_store_uri},
+            NIX_CONF_DIR  => $ctx->{builder}{nix_conf_dir},
+            NIX_STATE_DIR => $ctx->{builder}{nix_state_dir},
+            NIX_STORE_DIR => $ctx->{builder}{nix_store_dir},
+            RUST_LOG      => "hydra_builder=debug,info",
+        },
+    );
+
+    _wait_for($self->{ua}, "$self->{base_url}/status/machines", sub {
+        shift->decoded_content =~ /"hostname"/;
+    }) or die "Timed out waiting for builder to register\n";
+}


Very sus! we already have code to start the builder, we should not need new such code.

Ericson2314 · 2026-04-30T22:05:36Z

+sub _get_random_port {
+    my ($min, $max) = @_;
+    while (1) {
+        my $port = $min + int(rand($max - $min + 1));
+        my $sock = IO::Socket::IP->new(
+            LocalAddr => '::',
+            LocalPort => $port,
+            Proto     => 'tcp',
+            ReuseAddr => 0,
+        );
+        if ($sock) {
+            close($sock);
+            return $port;
+        }
+    }
+}


Should be shared with other htings

Ericson2314 · 2026-04-30T22:06:56Z

I think it would be best to be able to run the entire test suite (or at least the parts doing evaluation) with and without the new drv daemon.

That means that the new IFD and ad hoc jobs tests would require the new daemon, but every other test would run twice in both modes.

Ericson2314 · 2026-04-30T22:13:53Z

+        // unix:// ignores NIX_STORE_DIR, so NIX_REMOTE must include the logical store dir.
+        auto drvDaemonSocket = config->getStrOption("drv_daemon_socket");
+        auto nixStoreDir = config->getStrOption("nix_store_dir", "/nix/store");
+
        jobset.pid = startProcess([&]() {
+            if (!drvDaemonSocket.empty()) {
+                auto remote = "unix://" + urlEncodePath(drvDaemonSocket)
+                    + "?store=" + urlEncodePath(nixStoreDir);
+                setenv("NIX_REMOTE", remote.c_str(), 1);
+            }


I am not sure, but this feels like the wrong layer of abstraction. I forgot that hydra-evaluator was its own separate demon, not one of the "scripts".
What that means is that:

If the setting stays in the config, it should be holistic --- do we want evaluations to use the drv daemon or not? That is the actual policy choice here.

If we instead mess around with the underlying mechanisms, we could just set NIX_REMOTE for the entire hydra-evaluator from the outside, not within it. I.e. it is the responsibility of however the daemons set up and run to wire things up correctly.

Ericson2314 · 2026-04-30T22:15:11Z

+/* Encode path/query separators before embedding operator paths in a unix:// store URL. */
+static std::string urlEncodePath(const std::string & s)
+{
+    std::string out;
+    out.reserve(s.size());
+    for (unsigned char c : s) {
+        bool safe = std::isalnum(c)
+            || c == '-' || c == '.' || c == '_' || c == '~'
+            || c == '/' || c == ':';
+        if (safe) {
+            out.push_back(static_cast<char>(c));
+        } else {
+            char buf[4];
+            std::snprintf(buf, sizeof(buf), "%%%02X", c);
+            out.append(buf);
+        }
+    }
+    return out;
+}
+


Per b5e42d4#r3171126384 we might not even need this (because the environment variables are set externally and not in this C++, option 2).

If we got with option 1, I think we can replace this with a Nix function, not hand-roll it.

Ericson2314 · 2026-04-30T22:23:01Z

+        Logical store directory. Must match the store dir used by the
+        queue runner / builder; the value is also what the evaluator
+        appends as `?store=<dir>` when it points `NIX_REMOTE` at the
+        socket, since `unix://` connections ignore `NIX_STORE_DIR`.


I don't think that is true, I don't think they ignore NIX_STORE_DIR

But also, we much rather not rely on NIX_STORE_DIR and just use NIX_REMOTE, so maybe this fine.

Ericson2314 · 2026-04-30T22:24:51Z

+    routeEvaluatorIfds = lib.mkOption {
+      type = lib.types.bool;
+      default = true;
+      description = ''
+        When true, wire hydra-evaluator to this daemon and order it after hydra-drv-daemon.service.
+
+        Note: routing only takes effect when
+        `allowImportFromDerivation` is also enabled — otherwise
+        `nix-eval-jobs` rejects every IFD before the daemon can see it.
+      '';
+    };


I don't understand why this is a setting for hydra-drv-daemon itself. That seems wrong --- why does it care who is using it and for what purpose?

Ericson2314 · 2026-04-30T22:25:01Z

+    allowImportFromDerivation = lib.mkOption {
+      type = lib.types.bool;
+      default = false;
+      description = ''
+        When true, enable IFD evaluation in hydra.conf. This lets untrusted evals run builders, so it stays opt-in.
+      '';
+    };


Port hydra-evaluator from C++ to Rust, removing the last C++ executable from the project. The new implementation uses `sqlx` and `tokio`, matching the async patterns already established by hydra-queue-runner and hydra-builder. Build/packaging changes: - Remove all C++ dependencies from `subprojects/hydra/meson.build` - Add `subprojects/hydra-evaluator/package.nix` following the same per-binary pattern as `hydra-queue-runner` and `hydra-builder`: each Rust binary is its own derivation that builds against a fileset containing only its own crate, and `postPatch` strips the other Rust binary crates from the workspace `members` list so cargo doesn't try to load their (absent) manifests. Properly sharing the dep crates between these builds is left for later. - Add `evaluatorExecutable` option to the NixOS web-app module, matching how the queue runner and builder are discovered via their own modules - Clean up stale references in `hydra-tests/meson.build` and `dev-shell.nix` `JobsetRow` uses `i32` for `lastCheckedTime` and `triggerTime` to match the `INT4` Postgres schema (`sqlx` refuses to decode `INT4` into `i64`). Post-eval DB updates are wrapped in a transaction, as in the C++ original. The DBI parser supports `sslmode`/`sslrootcert`/`sslcert`/ `sslkey` that `libpq` understood natively in the C++ version; unknown parameters are warned about rather than fatal. Satisfy the crate's own `#![deny(clippy::pedantic)]` gate: - use `fs_err` per workspace `disallowed_methods` policy - replace `as`-casts with `try_from` (schema is `INT4`, saturate at `i32::MAX`) - collapse nested `if let` chains Co-authored-by: Jörg Thalheim <joerg@thalheim.io> Co-authored-by: Claude <noreply@anthropic.com>

Add hydra-drv-daemon, a nix-daemon protocol endpoint that turns build requests into ad-hoc Hydra Builds and waits for queue-runner completion. Read operations and store uploads are proxied to the upstream nix-daemon.

When drv_daemon_socket is configured, hydra-evaluator runs jobset evaluators with NIX_REMOTE pointing at hydra-drv-daemon. Add the NixOS service/module and dev Procfile wiring.

Add an end-to-end test stack for submitting a derivation through hydra-drv-daemon and building it through queue-runner/builder. The fixture is input-addressed because CA realisations are not surfaced yet.

amaanq force-pushed the drv-daemon branch from 7b70c4b to 7a57c0c Compare April 30, 2026 20:06

Ericson2314 reviewed Apr 30, 2026

View reviewed changes

Ericson2314 marked this pull request as draft May 2, 2026 20:39

amaanq force-pushed the drv-daemon branch 4 times, most recently from ad5b267 to 5f6e67f Compare May 4, 2026 21:26

Ericson2314 and others added 4 commits May 6, 2026 13:49

drv-daemon: ad-hoc Builds + queue-runner handoff

d00353c

Add hydra-drv-daemon, a nix-daemon protocol endpoint that turns build requests into ad-hoc Hydra Builds and waits for queue-runner completion. Read operations and store uploads are proxied to the upstream nix-daemon.

evaluator: route IFDs through hydra-drv-daemon

a2f6c5f

When drv_daemon_socket is configured, hydra-evaluator runs jobset evaluators with NIX_REMOTE pointing at hydra-drv-daemon. Add the NixOS service/module and dev Procfile wiring.

tests: imperative ad-hoc build via drv-daemon

badcd81

Add an end-to-end test stack for submitting a derivation through hydra-drv-daemon and building it through queue-runner/builder. The fixture is input-addressed because CA realisations are not surfaced yet.

Ericson2314 force-pushed the drv-daemon branch from 5f6e67f to badcd81 Compare May 6, 2026 17:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

drv-daemon: ad-hoc Builds + queue-runner handoff#1692

drv-daemon: ad-hoc Builds + queue-runner handoff#1692
amaanq wants to merge 4 commits into
NixOS:masterfrom
obsidiansystems:drv-daemon

amaanq commented Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026 •

edited

Loading

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 commented Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026 •

edited

Loading

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Ericson2314 Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

amaanq commented Apr 30, 2026

Motivation

Solution

Uh oh!

Ericson2314 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ericson2314 commented Apr 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ericson2314 Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ericson2314 Apr 30, 2026 •

edited

Loading

Ericson2314 Apr 30, 2026 •

edited

Loading