The portable toolshed, a reproducible specification for:

• packages (tools, utilities)
• dotfiles (configuration of tools)

... that works across any Linux or macOS and has a single-step, set-up/update procedure:

script/install.sh

NOTE: the packages built here are available in this cachix cache.

NOTE: this is my personal toolchain: I recommend you fork unless you want my workflow preferences overriding yours.

1. Homogenize tool availability and behavior across systems, normalize the entire work environment (or set up a blank machine) in one shot. No more:

   • checking what variant of sed you're running (GNU extensions?)
   • moving files around to certain systems because x tool is only installed on that machine.
   • sticking with e.g. an old Ubuntu 14.04 machine because it took you hours to get the set-up just right.

2. No more dealing with distro-specific package managers:

   • Remembering what was that package called in apt-get or yum or pacman or zypper or portage
   • installing MacPorts or Homebrew on macOS.

3. Make workflow tweaks rewarding: fix it once and it's fixed everywhere; rice your vimrc only once. No more:

   • separately tracking packages and versions for pip and bundler and cabal and npm etc ... on each machine
   • complex "toolchain set-up" shell scripts that break on every new OS release

4. Reasonably future-proof:

   • As Nix continues growing, this will work seamlessly on other POSIX systems, with no reconfiguration or fussing.
   • Worst-case scenario for moving to a new kernel+ABI is having to contribute to the so-called stdenv port.
   • The GNU people are on board, with guix.

1. One-shot set-up:

   # requires 'curl' at the very least
   script/install.sh

2. Try out the environment (requires Nix installation) without changing what you have currently:

   nix-shell --pure

3. Starting from scratch on macOS:

   # set up BASH as default shell
   chsh -s /bin/bash
   /bin/bash
   
   # install nix
   curl -L https://nixos.org/nix/install | sh
   source $HOME/.nix-profile/etc/profile.d/nix.sh
   
   # optional: set up cachix
   nix-env -iA cachix -f https://cachix.org/api/v1/install
   cachix use siriobalmelli-nixpkgs
   
   # get non-developer-tools git
   nix-env -A "nixpkgs.git"
   hash -r
   
   # clone this repo and install
   git clone https://github.com/siriobalmelli/toolbench.git
   cd toolbench
   script/install.sh default.nix

4. Tweaking/searching/installing packages:

   # get currently installed packages
   $ nix-env -q
   imagemagick-6.9.9-34
   jq-1.5
   less-530
   llvm-7.0.0
   # ...

   # uninstall a package
   nix-env --uninstall jq-1.5

   # get a list of available packages:
   $ nix-env -qaP
   nixpkgs._2048-in-terminal    2048-in-terminal-2017-11-29
   nixpkgs._20kly               20kly-1.4
   nixpkgs._2bwm                2bwm-0.2
   nixpkgs.go-2fa               2fa-1.1.0
   nixpkgs._389-ds-base         389-ds-base-1.3.5.19
   nixpkgs.pong3d               3dpong-0.5
   nixpkgs.all-cabal-hashes     70f02ad82349a18e1eff41eea4949be532486f7b.tar.gz
   nixpkgs.tiny8086
   # ...
   
   # for convenience, `script/install.sh` dumps this list to 'nix_env_avail.txt':
   $ grep gcc nix_env_avail.txt
   # ...
   nixpkgs.avrgcc               avr-gcc-8.2.0
   nixpkgs.gcc7                 gcc-wrapper-7.3.0
   nixpkgs.gcc8                 gcc-wrapper-8.2.0
   # ...

   # install a package
   nix-env --install nixpkgs.gcc8

5. Deal with configuration iterations/generations:

   # Listing the previous and current configurations:
   nix-env --list-generations

   # Rolling back to the previous configuration:
   nix-env --rollback

   # Deleting old configurations:
   nix-env --delete-generations [3 4 9 | old | 30d]

I am big fan of YCM for Vim.

Getting it working with Nix required writing a new .ycm_extra_conf.py to use nix-shell as a way to descry the includes seen by clang/gcc at compile-time.

To use, you will need to manually copy .ycm_extra_conf.py to the toplevel directory of your project:

# note that YCM expects '.ycm_extra_conf.py' but we maintain it as
# 'ycm_extra_conf.py' so that it's visible in our repo ;)
cp vim/ycm_extra_conf.py /path/to/my/project/.ycm_extra_conf.py

To better understand what's happening, keep in mind that the Nix people use the term wrapping a lot, without giving a straightforward definition. In a practical context, wrapping means "creating a runtime environment for something".

We can see the "runtime environment" for a compiler by running the preprocessor verbosely with some null input: clang -E -Wp,-v - </dev/null. This will print the list (and sequence) of locations where clang will look for header files:

1. From my environment:

   $ clang -E -Wp,-v - </dev/null
   #include "..." search starts here:
   #include <...> search starts here:
     /nix/store/fs62wczaxm8svvhwqsyv9nz4cca44lxa-clang-wrapper-7.0.0/resource-root/include
     /nix/store/kvdxajnlyisifi506ppbdpfycmcmsp6d-glibc-2.27-dev/include
   End of search list.

2. From inside nix-shell in some random project:

   $ nix-shell
   [nix-shell]$ clang -E -Wp,-v - </dev/null
   #include "..." search starts here:
   #include <...> search starts here:
     /nix/store/6j3xhdki32ni2admf1dhzvb2dgz0hl4c-dpkg-1.19.0.5/include
     /nix/store/8wg1n5bbhsv81wpyixcvxp547hz3q60x-libarchive-3.3.2-dev/include
     /nix/store/i6a6g9iph5sh5nd5vqf0r6yg1gaxv5g8-libbfd-2.30-dev/include
     /nix/store/dqr18y0x894fsrabwrlbrlhi870bnifc-elfutils-0.173/include
     /nix/store/0hvbn69zcavh8810kzqs1phv1l5f0lnh-liburcu-0.10.1/include
     /nix/store/n5qf4zfs6msxihh2xr8ndc7g9j8kc06v-compiler-rt-5.0.2-dev/include
     /nix/store/sjai00nfvq786fhwi5i0fw5b3qlrx572-clang-wrapper-5.0.2/resource-root/include
     /nix/store/kvdxajnlyisifi506ppbdpfycmcmsp6d-glibc-2.27-dev/include
   End of search list.

As you can see, the clang running inside nix-shell is not the same clang: it has been wrapped with the project dependencies. This is exactly the list we get by running nix-shell inside .ycm_extra_conf.py.

1. nixpkgs is pinned to a known working state (for both macOS and Linux) in https://github.com/siriobalmelli-foss/nixpkgs/tree/sirio (my fork).

2. Fork branches are organized as follows:

   • upstream/master : track nixpkgs
   • master : current stable version (reference this in Nix derivations)
   • development for upstream -(commit)-> fix/per-feature
   • fix/per-feature -(pull requests)-> upstream/master
   • fix/per-feture -(cherry-pick)-> sirio
   • unmergeable: vim-plugins/update.py etc -(commit)-> sirio
   • working config on Linux and macOS -(tag sirio_stable_YYYY_MM_DD)-> master

1. Originally forked from https://github.com/nmattia/homies and then customized, many thanks to Mr. Mattia. See his introduction blog post for an overview.

2. Many thanks to the Nix project and their community on IRC. Also the very helpful advice of jtojnar in this issue which helped me grok Nix a bit better.

3. The various Nix resources on the net:

• nix pills
• gentle nix introduction
• nix cheatsheet

There are a couple painpoints:

1. App selection and config file differences:

   • graphical app configs (eg Alacritty) will be different between Linux and Darwin
   • choice of window manager will be different based on OS
   • some systems should not have to install the full panoply of packages (eg no graphical apps, no heavy LaTeX stuff on a console-only linux VM)

   Explore how others are doing this:

   • Hugo Reeves: https://github.com/HugoReeves/nix-home/

2. Homogenization of dependencies: Need a way to force a single version of a dependency for all packages, eg

   • clang
   • gcc
   • python = python3 = python3.8

3. Remove boilerplate in config file generation, eg:

   • https://github.com/balsoft/nixos-config/blob/master/modules/workspace/i3blocks/scripts/battery.nix