“Does it really need to run as root?” When talking to system administrators of large supercomputers about installing Guix and having its build daemon run as root, this question would quickly come up—and rightfully so. We’re happy to announce that guix-daemon can now run without root privileges by taking advantage of Linux’s unprivileged user namespaces, a feature now available even on some of the most conservative supercomputers.

In this post, we look at how the daemon works when running as root, what changes when running as an unprivileged user, and what the security implications of each model are. We also discuss the opportunities offered by the new “rootless” mode.

The root of the privileges

Guix builds on the shoulders of Nix to provide isolated build environments, also referred to as hermetic builds. It is a foundation for several key properties:

• Controlled dependencies: a compilation process cannot mistakenly refer to a library or tool that happens to be on the build machine, thwarting a whole class of build issues one may encounter with package managers such as Brew or Spack.
• Sharing: since build processes are free from interference, if Alice builds GCC and Bob later asks for the same version and variant of GCC, Bob can reuse GCC as built by Alice since they know they would get the same result if they built it by themself. This was described by Eelco Dolstra in a seminal paper in 2005.
• Reproducible builds, or at least significantly reducing the chances that a build is not bit-for-bit reproducible: the source-to-binary correspondence can be verified by anyone.

The build daemon acts as a trusted third-party for all the users on a machine: it’s the only process that can modify the /gnu/store directory, which contains build results; guix commands merely ask it to perform actions on its behalf. When a command like guix build requests the compilation of a package, guix-daemon spawns the build process on behalf of that command.

To achieve the necessary level of isolation, the build daemon would run build processes with the authority of unprivileged build users, with a separate root directory (a chroot), and under separate namespaces. It is this combination of features that would guarantee that there can be no interaction between build processes and the outside world, or among build processes running concurrently.

Crucially, the ability to run processes under different build users meant that guix-daemon itself has to be running as root; the setuid call is only available to root (or more precisely to holders of the CAP_SETUID Linux “capability”) and there is no way around that.

Dropping privileges

The three security vulnerabilities found in guix-daemon in recent years allowed for privilege escalation: an unprivileged build process would be able to grant write access to the store or even to root-owned files to outside processes. Running guix-daemon without root privileges does not magically protect against vulnerabilities that would give unprivileged users write access to the store—thus potentially allowing an attacker to modify code anyone on the system might run—but it makes it impossible to directly escalate to root privileges.

For years we’ve been talking about the potential of Linux’s unprivileged user namespaces in this space, but also lamenting that they were disabled on most systems. Only recently did we start seeing support for unprivileged user namespaces on major supercomputers, which prompted us to reconsider that option.

If we add unprivileged user namespaces to our toolbox, the question is: can we achieve the same level of isolation as in the root model with guix-daemon running unprivileged? We knew it was feasible—in a similar domain, “rootless” podman became a thing in 2020—but the security implications of transitioning from one model to another are non-trivial. Think about it:

• Instead of having build processes running under different build users (the foundation that ensures they cannot interfere with one another), they would all run under the same user.
• Instead of having /gnu/store root-owned and thus only writable to guix-daemon, it would be writable by the user guix-daemon runs under and thus potentially writable for build processes as well.

The security mechanisms at play when guix-daemon runs unprivileged are very different, relying exclusively on Linux namespaces rather than on the well-understood UID-based access control.

The two-month long review process allowed us to identify a number of potential issues. A crucial part was ensuring that inputs to a build process—directories in /gnu/store containing the compiler, libraries, and so on—are mounted read-only inside the build environment and cannot possibly be remounted read-write. It took some careful reading of the fine lines of the manual to learn that mounts can be “locked together”—preventing build processes from unmounting or remounting them—and that we could ensure that mounts are indeed “locked” by checking the return value of umount. By studying the state of the art as far as escaping containers goes, fellow hacker Caleb Ristvedt found an interesting vulnerability that relied on the fact that while /proc/self/exe looks like a symlink, it really behaves like a hard link: it can be opened even if its target is outside the current chroot. Special care was needed to close this loophole.

There’s one privilege that guix-daemon does not drop when used on a distribution other than Guix System: the ability to change file ownership—the CAP_CHOWN Linux “capability”. This privilege is necessary to allow guix-daemon to create /var/guix/profiles/per-user/$USER, the user’s profile, and to change its ownership to the right user. We grant this capability via the systemd service file for guix-daemon. (On Guix System, CAP_CHOWN is unnecessary: user profiles are created upfront when the system boots or when it is reconfigured.)

Besides those security considerations, the unprivileged daemon introduces observable differences in the build environment that could in theory harm reproducibility. For example, build processes will now see that files under store are owned by themselves instead of being root-owned, and attempts to overwrite them or to write to the root file system will now fails with EROFS instead of EPERM. We don’t have hard data about this but our guess is that packages and their test suites are unlikely to rely on this behavior; all in all, it’s comparable to other aspects of the environment that the daemon does not control such as time, kernel, or file system details.

Trying it

So here we are: after much researching and hardening, we gained enough confidence to merge support for the unprivileged guix-daemon a few days ago. The daemon is now set up to run as an unprivileged user on distros that use systemd; it will soon be available as an option on Guix System.

This feature will be available in the upcoming Guix release. In the meantime, you can try it on the distro of your choice by doing a fresh installation, along these lines:

1. Grab a binary installation tarball from the snapshot download page;
2. Run GUIX_BINARY_FILE_NAME=guix-binary.tar.xz ./guix-install.sh, where guix-install.sh is the usual installation script.

And that’s it!

Migrating an existing installation to the unprivileged daemon on a distro other than Guix System is currently cumbersome: you’d have to manually upgrade Guix, copy the new guix-daemon.service systemd file, and change ownership of /gnu/store, /etc/guix, and a subset of /var/guix. We may provide a helper script in the future to automate this process.

Pivot to rootless

That guix-daemon does not require root privileges at all is a reassuring move, probably making it more amenable to installation in security-sensitive environments such as HPC supercomputers.

It will also be helpful in other contexts. For instance, using Guix in Docker (as happens on GitLab-CI) was unsatisfactory to say the least: you would have to either use Docker’s --privileged flag (yikes!) or pass --disable-chroot, effectively giving up on isolated builds. The unprivileged execution mode means that isolated builds are always available, without having to create separate build users. Incidentally, this is also useful for Guix’s own test suite, which is now able to perform builds in fully isolated environments as well.

Whether you’re a system administrator, a developer using Guix for continuation integration, or a tinkerer, we’d love to get your feedback!

Acknowledgments

Support for the unprivileged build daemon greatly benefited from the rigorous review of Caleb Ristvedt, and in particular from their thorough study of the all-too-often muddy namespace semantics and the loopholes they might leave open. Thanks also to Caleb for reviewing an earlier draft of this post.