Intro to Nix Channels and Reproducible NixOS Environment

This introduction assumes you have played with NixOS a bit, you know about content addressability and why it is important, and how Git repositories represent a distributed content addressed storage system.

Git and Github is used as the source control for all of NixOS and NixPkgs. Both NixOS and NixPkgs source code is located here: https://github.com/NixOS/nixpkgs. This means every package that is available via Nix is defined in that repository. This includes OS services and general software. Nix, the language interpreter and the package manager tool is however located here: https://github.com/NixOS/nix.

Channels is a rolling distribution endpoint that provides the latest builds of NixOS and Nixpkgs, and the Nix expression set corresponding to those builds. Because NixOS is just a particular build of NixPkgs that is geared towards running a full OS, we'll be using NixOS channels and not NixPkgs channels. The channels have names of the form:

• nixos-YY.MM
• nixos-YY.MM-small
• nixos-unstable
• nixos-unstable-small
• nixpkgs-unstable

We will ignore NixPkgs channels (nixpkgs-*) and only focus on NixOS channels (nixos-*) because the NixPkgs channels are intended to be used by non-NixOS users like other Linux distributions or Macintosh users. The exact artifacts that the channel points to will have their names detailed in this way: label-YY.MM.N.H-*. The label is is name of the artifact such as nixos, nixos-graphical, nixos-minimal. The N is the iteration number of the major version YY.MM. The H is a 7 character truncated Git hash. The * is the reamining metadata for the build artifact.

Channels are built from Git release branches on the official NixPkgs source repository: https://github.com/NixOS/nixpkgs. The branches are notated as release-YY-MM. On every commit to these branches, Hydra (the Nix build system) clones the source code and builds them while running tests. If these tests succeed, the successful branch commit is then propagated to a channels repository located at: https://github.com/NixOS/nixpkgs-channels. Hydra will also "release" them to the official channel endpoint at: https://nixos.org/channels/ . Once a commit is available at https://nixos.org/channels, it is guaranteed that the packages specified inside the corresponding Nix expressions will be built, and cached in the binary cache.

Let's see if these things match up (here we use httpie, pup and jq):

http https://nixos.org/channels/ | pup 'table tr:nth-child(n+4) td a attr{href}'

http 'https://api.github.com/repos/NixOS/nixpkgs/branches?per_page=100' | jq '[.[] | select(.name | startswith("release"))]'

http 'https://api.github.com/repos/NixOS/nixpkgs-channels/branches?per_page=100'

The released channel hashes (https://nixos.org/channels/) exactly match the channels repository branch hashes (https://github.com/NixOS/nixpkgs-channels).

However because Hydra takes time to build and verify release branches, the release branches at https://github.com/NixOS/nixpkgs will be sometimes ahead of the released channel branches and released channels.

Some extra things to note:

• The small channel variants are updated before the entire package set is built. This means the small channel variants will be up to date with the latest security changes and bug fixes compared to the non-small variants. This does mean that sometimes Nix will force a build from source rather than downloading pre-built packages.
• The nixos-unstable channel corresponds to the master branch at https://github.com/NixOS/nixpkgs. The master branch will sometimes be ahead of the channel, as the channel relies on Hydra to finish evaluation.
• There is no channel that corresponds to the staging branch.
• The staging branch at https://github.com/NixOS/nixpkgs is even more unstable than master. The staging branch is intended for changes that will cause mass-rebuilds of packages, and is supposed to be merged back into master every couple weeks.

So which source should we source our channels from? It depends:

• If you are developing on NixOS, you'll often need to make use of the latest commits or pull requests, in that case, you have to use branches from https://github.com/NixOS/nixpkgs.
• If you're just using NixOS and want to have a stable experience, you should use the official released channels endpoint at https://nixos.org/channels/
• If you're power user, but you still want a stable experience, you should use the Git release branches at https://github.com/NixOS/nixpkgs-channels.

The great thing about Nix, is that you can mix-and-match. Your system can be sourced from release branches, while cherry-picking changes directly from official nixpkgs repository, and even picking up forks, random commits and making use of them inside a Nix profile or a nix-shell.

However if you need to download the built ISOs for a given release, these should be downloaded from https://nixos.org/channels/, because the repositories don't host these artifacts. Another source for built artifacts is Hydra itself, but we'll not go into this yet.

It is at this point where we must differentiate the channels at https://nixos.org/channels/ and the Git repository sources from which those channels are built from. The minimal definition of a "channel" is a directory with 2 files: nixexprs.tar.xz and binary-cache-url. The nixexprs.tar.xz is the tarball that contains a default.nix which is what is actually used to evaluate Nix expressions. The binary-cache-url is just a text pointer to another location containing the cache for the Nix expressions.

The main utility of using these official channels over the sources in the Git repositories is so that you can know when the binary cache is ready to supply all the packaeges in the channel's Nix expression. Nix provides a command called nix-channel which allows you manage system and profile channels. The system channel is also the root user's channel.

Ultimately you can think of using Nix channels for every day usage, but you should be using the Git repository hashes as sources for your package sets when you care about reproducibility, which basically whenever you're developing software. On installation of a fresh NixOS system, there will be one channel the system is already subscribed to.

Running sudo nix-channel --list will show something like: nixos https://nixos.org/channels/nixos-17.03. The nixos name is just an alias, it allows you refer to multiple channels during package installation. (This is not as flexible as directly using the Git sources, because you need a proper channel structure including the binary-cache-url, and we don't have a valid binary cache for every single source commit hash.)

Having multiple channel aliases, allows you perform installations like nix-env --install --attr nixos.packagename or nix-env --install --attr mycoolchannel.packagename. This also means anybody can publish a channel, they just need to provide a binary cache as well.

We had to use sudo because by default in order to see the channel of the root user/system. But every user on a NixOS system can manage their own set of channels, where the channel aliases can override the system channel aliases.

# make sure you don't have `.` or `-` in the channel name!!!
# this is a bug: https://github.com/NixOS/nix/issues/1457
nix-channel --add https://nixos.org/channels/nixos-17.03-small nixosSmall1703
nix-channel --update nixosSmall1703
nix-env -iA nixosSmall1703.xmlstarlet

Adding a channel by itself does not download the channel expressions, you need explicitly run the update command. The channels are stored in /nix/store but referenced from ~/.nix-defexpr (this exists for both normal users and the root user). If you want to learn more about this, watch the contents of ~/.nix-defexpr/channels/ and ~/.nix-defexpr/channels_root/ while running the above commands.

The nix-channel command is quite limited, it does not tell you what channels are out of date, so you would need to run the --update command often to get the latest released iterations of each channel. You can find the exact version of your channel by looking at cat ~/.nix-defexpr/channels/nixosSmall1703/svn-revision or ~/.nix-defexprs/channels_root/nixos/svn-revision. But this is not an official supported technique.

You can remove channels with nix-channel --remove nixosSmall1703, and you don't need to run nix-channel --update.

Because of how channels work, we prefer to side-step this functionality and instead pin our NixOS system a particular commit hash within a released channel. This is what makes a NixOS system reproducible. We'll try to make sure that the packages we want to install are all from a single commit hash. Note that this is sometimes not possible when a package at a particular iteration or version is only available at a different commit hash, and you don't want to move your entire system to that commit hash. However with Nix, it's very easy to have packages from multiple different commit hashes cohabiting on the same system, and when you need isolation, this can be provided via Nix profiles or nix-shell.

While we could just import the desired Nix expression inside our configuration.nix, certain tools like nix-env relies on the ambient channel setup to install packages. The right way to solve this would be to make nix-channel support specific commit hashes and local directories containing Nix expressions, rather than only channel directories. This would make pinning a NixOS system much easier, and would seamlessly work with the existing nix-env expectations. There is however a workaround involving the utilisation of the Git content-addressed system and NIX_PATH. This workaround was discovered here: http://anderspapitto.com/posts/2015-11-01-nixos-with-local-nixpkgs-checkout.html ( There is an error with this workaround that is discussed below.)

# switch to super user
sudo --login
# we're going to create /nix/nixpkgs as our pinned nix expressions
rm --recursive --force /nix/nixpkgs
git clone https://github.com/nixos/nixpkgs /nix/nixpkgs
cd /nix/nixpkgs
git remote add channels https://github.com/nixos/nixpkgs-channels
git fetch --all
git checkout -B channels-nixos-17.03 channels/nixos-17.03
# /nix/nixpkgs will now be set to branch channels-nixos-17.03 and tracking channels/nixos-17.03

Now go into your configuration.nix, and set nix.nixPath = [ "nixpkgs=/nix/nixpkgs" "nixos-config=/etc/nixos/configuration.nix" ];.

Since the current NIX_PATH hasn't updated yet, we need to force a rebuild with the the new source for Nix expressions: nixos-rebuild -I nixpkgs=/nix/nixpkgs switch.

When you want to update you can just perform a git pull on the directory. These Git commands will also be helpful when you want to checkout a specific commit hash, or when you want to switch to a different channel source.

# show remote info
git remote show channels
# update all remotes
git fetch --all
# see all the branches that is available
git branch --all --verbose --verbose
# see what the remote provides along with pull requests to that remote
git ls-remote channels
# if you have added changes to your local branch, you can rebase your changes on top of any updates
git rebase channels/nixos-17.03 channels-nixos-17.03
# if you want to reset to upstream channel and drop any local changes (resets the working tree as well)
git reset --hard channels/nixos-17.03

Now we can remove the system channel and any user-channels as they will no longer be used:

nix-channel --remove nixosSmall1703
sudo nix-channel --remove nixos

All nix commands except the nix-env command will work. The nix-env currently only uses ~/.nix-defexpr. This issue (https://github.com/NixOS/nix/issues/993) asked to make nix-env use NIX_PATH or <nixpkgs> as a fallback if there are no channels installed. However this was only implemented in the new Nix UI project, and is not available on the current nix-env (1.11.15). The only way to make nix-env use NIX_PATH is to alias it with the option --file '<nixpkgs>'. This is safe as you can override the --file option by specifying it again. Once you do this, attribute path installation doesn't require a channel name. For example: nix-env --file '<nixpkgs> --install --attr 'hello'.

While this makes sense for general purpose use, for unattended NixOS servers where it doesn't make sense to use nix-env, there is no need to do all of this, and you can just directly import a content-addressed Nix expression set inside your configuration.nix.

At this point we can see how to acquire packages from different sources (including other commit hashes) and have them all coexist. There are 2 ways to do this. Through the imperative nix-env command that mutates your profile state, and through Nix expressions that you can inline into your configuration.nix or your nix-shell configuration files.

nix-env --file '<nixpkgs>'
nix-env --file '/package.nix'
nix-env --file '/directory/containing/default.nix/'
nix-env --file '/tarball/containing/default.nix/tarball.tar'
nix-env --file 'https://github.com/NixOS/nixpkgs/archive/master.tar.gz'
nix-env --file 'https://github.com/NixOS/nixpkgs/archive/74f22ff827d7c91fe26a62c69a53556a62ecc70c.tar.gz'
nix-env --file 'https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz'

Because /nix/nixpkgs is owned by root, if you run Git operations via sudo, you'll end up with permissions that don't allow usage by other users (due to the default umask used by Git when creating new files). You'll need to run these commands to reset to proper permissions. We need to make sure that all directories are at least can be readable and executable by all users, and that all files are at least readable by all users.

# some files are left executable inside /nix/nixpkgs
# they are just build support files
sudo chmod --recursive u=rwX,g=,g=rX,o=,o=rX /nix/nixpkgs

There may be a way to avoid these permission problems in the future.

As you can see we rely on Github to expose specific commit hashes as tarballs that our nix-env can consume. This makes it convenient to install packages from different branches, forks and even pull requests.

The same thing can be done inside a Nix expression such as our configuration.nix. The key primop is import. This command expects either a directory containing default.nix or a Nix file. If we intend to acquire an expression set remotely, we just need to use fetchTarball to expose the Nix expression. Note that nixpkgs based expressions is exported as a function, which means you can apply an empty attribute set to it before using it. The attribute set argument is intended to provide various flags into the Nixpkgs attribute set.

let
  pkgs-env = import <nixpkgs> {};
  pkgs-my = import ./package.nix;
  pkgs-dir = import ./packages/containing/default.nix/;
  pkgs-unstable = import (fetchTarball http://nixos.org/channels/nixos-unstable/nixexprs.tar.xz) {};
in
  {
    environment.systemPackages = [
      pkgs-env.packageAttrName
      pkgs-my.packageAttrName
      pkgs-dir.packageAttrName
      pkgs-unstable.packageAttrName
    ];
  }

Any path specified with angle brackets such as <nixpkgs> utilises the NIX_PATH environment variable to find files. We just set our NIX_PATH to nixpkgs=/nix/nixpkgs, which means <nixpkgs> just resolves to /nix/nixpkgs. However if we instead had <something/subthing>, this would try to look for something/subthing in each of the paths listed in the NIX_PATH. This is how it works:

NIX_PATH = nixpkgs=/nix/nixpkgs
<nixpkgs> = /nix/nixpkgs

NIX_PATH = nixpkgs=/nix/nixpkgs
<nixpkgs/lib> = /nix/nixpkgs/lib

NIX_PATH = nixpkgs=/nix/nixpkgs
<nonexistent> = Error

NIX_PATH = /somedir/containing/something/:nixpkgs=/nix/nixpkgs
<something> = /somedir/containing/something/something

You can find out ahead of time whether Nix will find an angular path by using: nix-instantiate --eval --expr '<something>'.

The same techniques can be further applied in your shell.nix when developing inside an isolated Nix shell, you should make sure your software environment is reproducible.

However if you are developing a package for the purpose of inclusion into official Nixpkgs, after you've done with development, if you are dependent on packages that is already available in one of the Nixpkgs iterations, you should rely on the ambient Nixpkgs provided through the pkgs attribute. This ensures that packages within a single Nixpkgs package set work in harmony with each other, and reduce the number of unnecessary duplicated dependencies. We have found that within the Nixpkgs source, there are times when multiple versions of a particular package is required by different downstream packages, this is when the particular iteration of Nixpkgs will often contain multiple versions of the same package. However, as soon as these conflicts are fixed, either by the upstream of the Nixpkgs maintainers, the older versions of the package will be deprecated and removed from future iterations of NixPkgs.

One possible downside is that you won't get automatic security updates with this approach. However this depends on your priorities. When in the act of developing software, reproducibility matters more. When in the act of maintaining/operating software, automatic security updates matter more. You can get the best of both worlds by building a system to monitor security issues relevant to your software dependency graph, and trigger updates then builds then tests before finally deploying.

---

UPDATE: We have found a way to avoid needing to apply permission changes after updating your /nix/nixpkgs. You do this by applying a temporary umask change. The reason is that the permission problems applies to new files being created by git.

# we use a subshell to make umask change temporary
(umask 022 && sudo git -C /nix/nixpkgs checkout channels-nixos-17.09)
(umask 022 && sudo git -C /nix/nixpkgs pull)

The above can be automated by changing sudo umask options: https://unix.stackexchange.com/a/278817/56970

Also, when you are using sudo while doing git operations requiring authentication. You may require sudo -E or sudo --preserve-env to preserve your SSH or GPG environment.