Nix, NixOS and NixPkgs allows us to create project-specific development environments with project-specific dependencies (this usually means things like a C project, or a Python project... etc).

The way this is done is different for every language community within the Nix ecosystem. The most well developed patterns would be the C/C++, Haskell and Python community, other language communities tend to be smaller and has less documentation. This article serves as an introduction to using Nix for developing projects in different languages that we have worked with. This means we will only focus on shell.nix.

Remember that Nix is a turing complete purely functional language, and so it has real functions. Defining a project environment within Nix is done via functions. The most common function utilised for this purpose is stdenv.mkDerivation. However, various language communities within Nix has wrapped this function in higher order functions to provide further automation to deal with the environment configuration that different language toolchains expect.

First decide whether your project is an "application" or a "library". If it's an application, you need a default.nix and a shell.nix. But if it's a library you only need a shell.nix. The default.nix that is created may is generally not the same as the default.nix that gets committed into the mainline nixpkgs repository. Your project's default.nix should be a self-contained expression that can be used to build and install your project. Whereas the default.nix committed into the mainline nixpkgs needs to be compatible with the rest of the nixpkgs package set in order to merge into a "harmonious" package set. In practice this means the default.nix in your project will use a pinned nixpkgs package set, whereas the default.nix in nixpkgs will expect a pkgs parameter to be passed in.

The default.nix will be utilised when you run nix-build. The shell.nix will be utilised when you run nix-shell.

The shell.nix defines a development environment, so it is usually simpler than the default.nix. The default.nix needs to also define installation locations.

When encountering dependencies not available under nixpkgs, you have 2 choices. Either write a nix expression for that package, or utilise a language-specific package manager. In our examples below, we will utilise language-specific package managers within our shell.nix in order to create environments that work even under non-Nix environments. However, for dependencies that rely on system libraries, we'll generally install them via Nix, rather than doing compilation without or nix-shell. To make this work elegantly, we want to make sure we are not specifying dependencies twice, and some language-specific package managers are able to recognise dependencies already installed by Nix.

Nix 2.0 has the ability to verify hashes when using the builtins.fetchTarball function with the sha256 property. To know what hash you want to use. Run nix-prefetch-url --unpack https://package-you-want-to-use. In our examples, we utilise this parameter. However, to make the expressions work under 1.x, remove the sha256 parameter.

Often language specific dependencies do not appear in the online package search engine https://nixos.org/nixos/packages.html. To actually find available packages, you can either look at the nixpkgs source code, or use nix-repl with :l <nixpkgs> and then accessing the package set relevant to your language. The same package set may be aliased to multiple names. Some names are more general than other names. While the names may seem ambiguous, within a content-addressed package set, what these names map to is deterministic. In subsequent nixpkgs versions, the names may point to new package sets.

If you are making use of pinned nixpkgs following our previous tutorial: https://matrix.ai/2017/03/13/intro-to-nix-channels-and-reproducible-nixos-environment/ You can acquire the hash of your current pinning via: git -C /nix/nixpkgs rev-parse HEAD.

The package set for C in nixpkgs is itself!

Here is an example shell.nix for a C project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/084445b8f38ff8196f4b3f16d0ad0e79aa88dcbc.tar.gz;
    sha256 = "0jqxx3csxbs32ijn8w6cbd9c3l9vvsjz57rqsyp44dgg37xxx00i";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "c-project";
    buildInputs = [ autoreconfHook ];
  }

Out of all the language environments, C has the best support. The stdenv.mkDerivation already brings in the standard C toolchain.

In this particular example, we've used autoreconfHook as a buildInput. You only need this if your project uses Autotools. If you use CMake, you would instead bring in cmake.

JavaScript

The package set for JavaScript in nixpkgs is: nodePackages.

Here is an example shell.nix for a JavaScript project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/00e56fbbee06088bf3bf82169032f5f5778588b7.tar.gz;
    sha256 = "15pl5p3f14rw477qg316gjp9mqpvrr6501hqv3f50fzlcxn9d1b4";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "javascript-project";
    buildInputs = [ nodejs-8_x flow ];
  }

Note how we are utilising nodejs-8_x. This is just an alias. What the alias means depends on the content addressed package set. We are also bringing in flow which is a JavaScript type checker.

Within this project we can use npm freely just like normal. So that's where we will put all our dependencies.

For most projects, this is enough. However, some projects require building C++ extensions. To do this we need to find the toolchain required to build the extension. The 3 common choices are node-gyp, node-gyp-build and node-pre-gyp. Look at the package.json of the package you're trying to install to find out which one is being used. Once you know this, you can add the relevant toolchain to the buildInputs.

# you don't actually need all 3 gyp toolchains, this is just an example
buildInputs = [
  nodejs-8_x
  flow
  nodePackages_8_x.node-gyp
  nodePackages_8_x.node-gyp-build
  nodePackages_8_x.node-pre-gyp
];

In some cases, you may also need python. It all depends on what kind of package you are installing.

The resulting project works in a Nix environment, and also works outside of a Nix environment.

Python

The package set for Python in nixpkgs is: python2Packages, python3Packages.

Here is an example shell.nix for a Python project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/630dbfe672164eeaf4bc822226cbb3ad7c1b0805.tar.gz;
    sha256 = "0xvj0z928mzcs56hindxw608a6jm1fvsi2ap5r7m4a0plnrcwx90";
  }) {}
}:
  with pkgs;
  python35Packages.buildPythonApplication {
    name = "python-project";
    buildInputs =  (with python35Packages; [
      numpy
      gdal
      scikitimage
      tensorflowWithCuda
      (matplotlib.override { enableQt = true; })
    ]);
    shellHook = ''
      echo 'Entering Python Project Environment'
      set -v

      # extra packages can be installed here
      unset SOURCE_DATE_EPOCH
      export PIP_PREFIX="$(pwd)/pip_packages"
      python_path=(
        "$PIP_PREFIX/lib/python3.5/site-packages"
        "$PYTHONPATH"
      )
      # use double single quotes to escape bash quoting
      IFS=: eval 'python_path="''${python_path[*]}"'
      export PYTHONPATH="$python_path"
      export MPLBACKEND='Qt4Agg'

      set +v
    '';
  }

This one is more sophisticated because it shows how to get Tensorflow with CUDA, and matplotlib rendering with Qt4.

Because pip by default installs into a global directory, we have to set PIP_PREFIX to a Project local directory. This directory should then be ignored by .gitignore.

When using pip, it actually is aware of dependencies installed via Nix. So in this case, if you later try to install a package that depends on numpy, pip will not reinstall numpy. Unless of course that dependency has a constraint that is not met by the numpy that you installed. The point is you can still write a normal setup.py and requirements.txt, and the project is still usable by non-Nix developers.

Haskell

The package set for Haskell in nixpkgs is: haskellPackages.

Here is an example shell.nix for a Haskell project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/8b1cf100cd8badad6e1b6d4650b904b88aa870db.tar.gz;
    sha256 = "1p0xxyz30bd2bg0wrfviqgsskz00w647h0l2vi33w90i42k8r3li";
  }) {}
}:
  with pkgs;
  haskell.lib.buildStackProject {
    name = "haskell-project";
    buildInputs = [];
    shellHook = ''
      echo 'Entering Haskell Project Environment'
      set -v

      alias stack="\stack --nix"

      set +v
    '';
  }

This example is really simple. Because here we are relying on stack to bring in all the Haskell dependencies similar to how we setup a shell.nix for JavaScript.

The usage of haskell.lib.buildStackProject wraps stdenv.mkDerivation and adds all these features: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/generic-stack-builder.nix

When using --nix flag on stack, it just makes sure that stack will utilise the GHC supplied by the haskell.lib.buildStackProject, instead of acquiring its own GHC, which would be a waste of time. All other stack commands work normally.

Becareful with the usage of the alias here. If you have scripts (like a Makefile) inside that call stack, they may not have --nix applied. If so something will go wrong. It would be better to have some sort of environment variable instead, but I'm not aware of it.

An alternative to using the alias is to set:

nix:
  enable: true

In your Project's stack.yaml or in ~/.stack/config.yaml. If you are expecting that your projects will be used by people not using Nix, then it makes sense not to put it in the Project's stack.yaml. However instead you can then put it in ~/.stack/config.yaml.

When using stack if a dependency doesn't build, that could be because it's missing a C library. The most common is probably zlib. If so, just put it into your buildInputs.

Emscripten

The package set for Emscripten in nixpkgs is: emscriptenPackages.

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/084445b8f38ff8196f4b3f16d0ad0e79aa88dcbc.tar.gz;
    sha256 = "0jqxx3csxbs32ijn8w6cbd9c3l9vvsjz57rqsyp44dgg37xxx00i";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "emscripten-project";
    buildInputs = [
      emscripten
      cmakeCurses
      pkgconfig
      python2
      nodejs
      flow
    ];
    shellHook = ''
      EMSCRIPTEN_ROOT_PATH='${emscripten}/share/emscripten'
      EMSCRIPTEN='${emscripten}/share/emscripten'
    '';
  }

At this moment we are not aware of any function which wraps stdenv.mkDerivation that deals with emscripten requirements. Preferably this can either be stored in a package hook when you bring in emscripten or it can be placed into a special function. This means at this point if you bring in an emscriptenPackages, it isn't automatically registered. You have to find the relevant environment variable to configure in your shellHook.

PHP

The package set for PHP in nixpkgs is: phpPackages.

Here is an example shell.nix for a PHP project:

{
  pkgs ? import (fetchTarball {
    url = https://github.com/NixOS/nixpkgs-channels/archive/084445b8f38ff8196f4b3f16d0ad0e79aa88dcbc.tar.gz;
    sha256 = "0jqxx3csxbs32ijn8w6cbd9c3l9vvsjz57rqsyp44dgg37xxx00i";
  }) {}
}:
  with pkgs;
  stdenv.mkDerivation {
    name = "php-project";
    buildInputs = [
      php71
    ] ++ (with php71Packages; [
      composer
    ]);
  }

Here we rely on composer to bring in all the PHP dependencies. So it's just like JavaScript.

Database Services

Application projects often might rely on a database service. You can use shell.nix shellHook to help set this up, so that each project can have its own database running within the shell. To make this work without adding in container/network namespaces, we rely on unix domain sockets. This allows each database process to only communicate with a project local unix domain socket.

Here is an example using MySQL and the Flyway migrations system:

shellHook = ''
  echo 'Entering MySQL Environment'
  . ./.env
  set -v

  alias mysql="\mysql --socket='$(pwd)/.mysql/mysql.sock' "$DB_DATABASE""
  alias mysqladmin="\mysqladmin --socket='$(pwd)/.mysql/mysql.sock'"
  alias mysqld="\mysqld \
    --datadir="$(pwd)/.mysql" \
    --socket="$(pwd)/.mysql/mysql.sock" \
    --bind-address="$DB_HOST" \
    --port="$DB_PORT""

  alias flyway="\flyway \
    -user="$DB_USERNAME" \
    -password="$DB_PASSWORD" \
    -url="jdbc:mysql://$DB_HOST:$DB_PORT/$DB_DATABASE" \
    -locations=filesystem:$(pwd)/migrations";

  set +v
'';

Notice the .env file which stores all the environment variables. So you can use shell.nix as a dotenv replacement.

Remember to set up initialise your database before you start using it!

rm -rf .mysql && mkdir .mysql mysqld --datadir="$(pwd)/.mysql" --initialize-insecure

launch your database without networking!

mysqld --skip-networking &

Again beware of using aliases, it is actually better to use environment variables when possible as they will carry over to any scripts you run.

Here's a bonus using PostgreSQL and PostGIS:

{
  pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/49a0ecc49b491c1f1a6f329d3d4fc9cf559b18aa.tar.gz) {}
}:
  with pkgs;
  let
    pg = (pg:
      buildEnv {
        name = "postgresql-and-plugins-${(builtins.parseDrvName pg.name).version}";
        paths = [ pg pg.lib (postgis.override { postgresql = pg; }) ];
        buildInputs = [ makeWrapper ];
        postBuild = ''
          mkdir -p $out/bin
          rm $out/bin/{pg_config,postgres,pg_ctl}
          cp --target-directory=$out/bin ${pg}/bin/{postgres,pg_config,pg_ctl}
          wrapProgram $out/bin/postgres --set NIX_PGLIBDIR $out/lib
        '';
      }
    );
  in
    stdenv.mkDerivation {
      name = "project-with-postgres-postgis";
      buildInputs = [ (pg postgresql) ];
    }