Skip to content

nixos/etc-overlay: implement the (re)mount logic in rust in nixos-init and improve mutable state behaviour - #533343

Open
r-vdp wants to merge 3 commits into
NixOS:staging-nixosfrom
r-vdp:rvdp/nixos-init-etc-overlay
Open

r-vdp wants to merge 3 commits into
NixOS:staging-nixosfrom
r-vdp:rvdp/nixos-init-etc-overlay

Conversation

@r-vdp

@r-vdp r-vdp commented Jun 19, 2026 •

Copy link
Copy Markdown
Contributor

Things done

Rewrite the overlay /etc mounting logic in Rust as part of nixos-init, replacing the existing bash implementation.
Because:

  1. Using the linux mount API directly is much simpler than going through the util-linux wrappers from shell
  2. On recent kernels we can mount the erofs metadata image directly from a file, without setting up a loop device
  3. No more temporary mount points to create and clean up. The new mount API lets us build the overlay as a detached mount held by an fd, which is cleaned up automatically if we crash, and then move it into place in one step
  4. No need for the standalone mount_beneath C wrapper, we set the mount flags directly
  5. Once the activation script is gone, this can run directly from nixos-init activate

This also fixes the mutable overlay switch logic.
Previously, switching between generations reused the live upperdir and combined it with a new lowerdir.
The kernel warns about this because sharing the upperdir of a mounted overlayfs is undefined behaviour, and the upperdir state can get out of sync with the lowerdir.
We also touched the live upperdir in place, so a crash mid-switch could leave it in a half-cleaned state.

In practice this meant that once a managed /etc file or directory was deleted or modified at runtime, overlayfs left whiteouts, opaque markers, or copied-up files in the upperdir that kept shadowing the declaratively managed version across rebuilds, with no practical way for a user to get back to it or even see what happened (see for instance #505475).
The old cleanup only handled the opaque case.

The new behaviour on live switch:

  1. Take a hardlink snapshot of the upperdir
  2. Reconcile the snapshot against the new lowerdir: remove whiteouts, opaque markers, and copied-up files that shadow nixos-managed entries
  3. Mount the new overlay against the snapshot, slide it beneath the current /etc, and detach the old mount
  4. Swap the old and new upperdirs using RENAME_EXCHANGE and delete the old ones.

A crash at any point leaves the original upperdir untouched.
In initrd we keep reconciling in place since nothing has the upperdir mounted at that point.

Fixes #505475

@r-vdp
r-vdp requested review from RossSmyth and nikstur June 19, 2026 16:22
nixpkgs-branch-check[bot]

This comment was marked as outdated.

@nixpkgs-ci
nixpkgs-ci Bot requested a review from a team June 19, 2026 16:28
@nixpkgs-ci nixpkgs-ci Bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-nixos-tests This PR causes rebuilds for all NixOS tests and should normally target the staging branches. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 6.topic: systemd Software suite that provides an array of system components for Linux operating systems. labels Jun 19, 2026
@r-vdp
r-vdp changed the base branch from master to staging-nixos June 19, 2026 16:37
@nixpkgs-ci nixpkgs-ci Bot closed this Jun 19, 2026
@nixpkgs-ci nixpkgs-ci Bot reopened this Jun 19, 2026
@r-vdp r-vdp changed the title Rvdp/nixos init etc overlay nixos/etc-overlay: implement the (re)mount logic in rust in nixos-init and improve mutable state behaviour Jun 19, 2026
@nixpkgs-branch-check
nixpkgs-branch-check Bot dismissed their stale review June 19, 2026 16:43

Review dismissed automatically

@RossSmyth

Copy link
Copy Markdown
Contributor

Nice! I'll test it either tonight or sometime this weekend.

@RossSmyth RossSmyth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is acting as I expect it to:

  1. run0 rm /etc/nixos/nixpkgs
  2. nixos-rebuild .......
  3. ls /etc/nixos

and the symlink is back.

Then it also didn't mess with NetworkManager, which is another worry I have with /etc things, as if I enable an immutable /etc, nm doesn't work as it cannot place files in /etc.

Same thing when I restart my computer. It places the /etc/nixos/nixpkgs symlink back, and wifi still works.

@nixpkgs-ci nixpkgs-ci Bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Jun 20, 2026
@nixpkgs-ci nixpkgs-ci Bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 7, 2026
Comment thread pkgs/by-name/ni/nixos-init/src/etc_overlay.rs Outdated
@r-vdp
r-vdp force-pushed the rvdp/nixos-init-etc-overlay branch 2 times, most recently from 3198d46 to c980646 Compare July 21, 2026 21:48
@nixpkgs-ci nixpkgs-ci Bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. and removed 2.status: merge conflict This PR has merge conflicts with the target branch labels Jul 21, 2026
@RossSmyth

Copy link
Copy Markdown
Contributor

I've not tested it, but I've come across another issues where switching to resolved does not symlink /run/systemd/resolve/stub-resolv.conf to /etc/resolv.conf. My guess is that this PR would fix that as it is a similar issue.

@nixpkgs-ci nixpkgs-ci Bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 19, 2026
r-vdp added 3 commits October 25, 2026 11:42
Replace the find-etc and clear-etc-opaque entrypoints with two new
ones that call into a shared setup_etc():
initrd-etc-overlay (discovers the layout from bootspec) and
mount-etc-overlay (takes it on the CLI for activation/nixos-enter).
The overlay and erofs metadata are built as detached mounts, with
file-backed erofs and fd-based lowerdir+ on recent kernels and
losetup / path-based lowerdir as fallbacks.
When /etc is already mounted, the new overlay is slid beneath it and
the old one is detached.

The upperdir reconciler now also removes whiteouts and copied-up
files that shadow nixos-managed entries, so deleting or replacing a
managed /etc entry at runtime no longer hides it permanently.

Fixes NixOS#505475
Reconciling the upperdir of the live /etc overlay during switch is
undefined behaviour since we mess with the state of a live overlayfs.
We already get a kernel warning in the logs because of the shared upperdir
between the two overlayfs mounts.
It also risks losing user replacements of managed files if activation
crashes before the new mount is revealed.

Instead, take a hardlink-tree snapshot at /.rw-etc/upper.next,
reconcile and mount against that, and promote it with RENAME_EXCHANGE
after the new overlay is in place. A crash at any point leaves the
canonical /.rw-etc/upper untouched. The initrd path keeps reconciling
in place since nothing has the upperdir mounted there.
@r-vdp
r-vdp force-pushed the rvdp/nixos-init-etc-overlay branch from c980646 to 84ed6c1 Compare October 25, 2026 14:18
@nixpkgs-ci nixpkgs-ci Bot removed 2.status: merge conflict This PR has merge conflicts with the target branch 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. labels Oct 25, 2026
@RossSmyth

Copy link
Copy Markdown
Contributor

I do not use mutable etc anymore, but it would still be good to see this. It did work back when I did have a mutable etc, but I would like to get some eyes on some of the changes besides just me before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: systemd Software suite that provides an array of system components for Linux operating systems. 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-nixos-tests This PR causes rebuilds for all NixOS tests and should normally target the staging branches. 12.approvals: 1 This PR was reviewed and approved by one person.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nixos/system/etc/etc: Specified symlinks not put in /etc bindmount

3 participants