Monthly Archives: July 2014

FNIStash r1.5 released!

Posted on by

I put up a new version of FNIStash today. There aren’t any new features, but successfully recognizing items in the shared stash without any error has been dramatically improved. My wife and I have been using this version for a while and only rarely, rarely come across an item FNIStash has problems handling. In fact, the only item I know of right now that has a problem is Landchewer…

If you’ve been holding off on trying out FNIStash because it wasn’t reliable, I think you’ll be a lot happier with r1.5 than previous releases.

Remember, if you have a question or problem, don’t be afraid to contact me or leave a comment! I like to think I’m pretty responsive.

Setting Up a Haskell Project on NixOS

Posted on by
This entry is part 6 of 6 in the series Setting Up Haskell Development on NixOS Linux from Scratch

Previously, we looked at how to add new packages to our nixpkgs expression tree. Often, the reason we are adding new packages is that we want to use them for Haskell development! This installment will explore starting a new Haskell project.

Again, we’re going to be using some custom nix commands that we set up in previous posts of this series, so if you’re jumping in here, you might want to back up a few installments if you find yourself getting lost.

I’m going to be using the janrain snap tutorial as the basis for this installment, but much of this exploration is getting set up to do the tutorial, so there is not much overlap.

Set Up Project Files

First, let’s set up the basic project files we’re going to be using for our exploration.

Basic Snap Setup

Run the following commands to set up a new projectomatic directory and initialize it using cabal init and snap init. For the cabal init call, you can accept all the defaults if you want; most of the files generated with that call will be overwritten by snap init anyway. When asked, you want to choose 2) Executable.

#!bash
mkdir projectomatic
cd projectomatic
cabal init
snap init

If you encounter a snap: command not found message, then you need to install snap into your user environment. If you already have the package, installing it will just make it active for the user environment so you can call snap init. Otherwise, installing will both build it and make it active.

#!bash
nix-install haskellPackages.snap

The project directory will be initialized with a projectomatic.cabal file, among others. Feel free to modify the values set by default in the cabal file. I’m going to leave the defaults as is for this tutorial.

Sanity Check: In your projectomatic directory, you should now have projectomatic.cabal, Setup.hs, and the directories log, snaplets, src, and static.

Generate default.nix

Just like we used the cabal2nix utility to automatically generate nix expression for hackage packages, we can also use cabal2nix to generate expressions for our new projectomatic project.

#!bash
cabal2nix projectomatic.cabal --sha256 blah > default.nix

Check out the new default.nix and you’ll see a nice, clean expression for the new projectomatic package. We want to change the sha256 = "blah" to src = ./., but other than that, the expression is similar to the ones we saw previously.

We can’t run it just yet, though. This expression is generated so that it can be integrated into the haskellPackages environment of nixpkgs. Since our project will be distinct from haskellPackages, we need to do some additional customization.

The first issue is with the expression inputs:

#!nix
{ cabal, heist, lens, MonadCatchIOTransformers, mtl, snap, snapCore
, snapLoaderStatic, snapServer, text, time, xmlhtml
}:

This looks just like the previous nix expressions we encountered, so what’s the problem? When we incorporated new nix expressions into the nixpkgs expression tree, expressions for dependent Haskell packages were defined in the same haskellPackages scope. Thus, when Nix needed the input cabal expression in order to put the cabal dependency on the path, it could do it because the cabal expression was in the same scope.

On the other hand, here we are defining a nix expression outside of that scope, so Nix doesn’t know what the cabal dependency means. In fact, if we try to run nix-build for projectomatic, we’ll see this:

#!bash
[dan@nixos:~/Code/projectomatic]$ nix-build
error: cannot auto-call a function that has an argument without a default value (`cabal')

To fix this problem, we need to specify in our default.nix expression where the dependent expressions are defined. We’ll do this by redefining the expression to take a single haskellPackages input argument that has a default value defined as (import <nixpkgs> {}).haskellPackages), which is the scope we want!

#!nix
{ haskellPackages ? (import <nixpkgs> {}).haskellPackages }:
let 
  inherit (haskellPackages) cabal heist lens MonadCatchIOTransformers mtl snap snapCore
  snapLoaderStatic snapServer text teim xmlhtml
  ;

in cabal.mkDerivation (self: {

The term <nixpkgs> instructs Nix to look for expressions beginning with $NIX_PATH/nixpkgs. Since my NIX_PATH has only the prefix that I set in my .bashrc file, the resolved file path is my desired local directory /home/dan/Code/nixpkgs.

#!bash
[dan@nixos:~/Code/projectomatic]$ echo $NIX_PATH
/home/dan/Code:nixos-config=/etc/nixos/configuration.nix

Using nix-shell

Now we have all the files we need for a complete (but minimal) snap default project. A lot of our effort was spent tweaking the new default.nix to work with our new project. Surely there must be some way to check that it’s correct.

A Quick Reminder about Expressions

Remember the purpose of nix expressions.

  1. Define all system inputs to a package
  2. Define the build steps for the package

When a nix expression is run, Nix configures something of a private system for building the package. The private system is configured to only have the dependencies of the nix expression. If a required dependency is left out of the expression inputs, the package will fail to build.

nix-shell Command

The nix-shell command acts as a way to inject a CLI into this private system. Within the nix-shell, the user can build expressions and run commands as if they are being run as part of an expression. Let’s try it out

#!bash
[dan@nixos:~/Code/projectomatic]$ nix-shell
error: Package ‘projectomatic-0.1’ in ‘/home/dan/Code/projectomatic/default.nix:18’ has an unfree license, refusing to evaluate. You can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this. If you use Nix standalone, you can add
  { allowUnfree = true; }
to ~/.nixpkgs/config.nix.

Whoa, what happened here? By default, Nix won’t evaluate expressions that have an unfree license, which, if you accepted all the defaults like I did, is what we have for our project. By following the instructions given in the error, we can allow unfree packages to be built. However, this will allow unfree packages everywhere, and we just want to build this one unfree package.

An unmentioned way to enable unfree packages is the NIXPKGS_ALLOW_UNFREE environment variable. We can temporarily enable unfree packages, build the one we want, and then disable it again to return us back to our starting point. I added this to my .bashrc file (along with nix-install and nix-search) to make it easy to drop into an unfree nix-shell.

#!bash
nix-shell-unfree() {
  FLAGSAVE=$NIXPKGS_ALLOW_UNFREE;
  echo "Opening shell with NIXPKGS_ALLOW_UNFREE=1. Original setting of $FLAGSAVE will restored on exit.";
  export NIXPKGS_ALLOW_UNFREE=1;
  nix-shell;
  echo "Restoring NIXPKGS_ALLOW_UNFREE=$FLAGSAVE ...";
  export NIXPKGS_ALLOW_UNFREE=$FLAGSAVE;
}

Now our nix-shell-unfree command will put us in a nix shell.

#!bash
[dan@nixos:~/Code/projectomatic]$ nix-shell-unfree
Opening shell with NIXPKGS_ALLOW_UNFREE=1.  Original setting of 0 will restored on exit

[nix-shell:~/Code/projectomatic]$ exit
exit
Restoring NIXPKGS_ALLOW_UNFREE=0 ...

Try entering a nix-shell-unfree and running nix-build. Did the build work without error? Depending on your dependency versions, maybe it did. For me, I get the following:

#!bash
Setup: At least the following dependencies are missing:
lens >=3.7.6 && <4.2

Looks like I still have some work to do. At first, I started setting up a new local expression for a different lens version, but after getting it set up, I found that snap requires lens 4.2, even though the cabal file written by by snap init requires less than 4.2! I am chalking this up to a bug and manually edited the projectomatic.cabal file to use lens <= 4.2. After that, nix-build works without issue. Note that I also tried keeping the cabal file unmodified and using jailbreak=true, but that had no effect as far as I could tell.

Running the Default Application

We have successfully overcome several obstacles in getting the default, (mostly) unmodified snap project template to compile. Now let’s run it.

When using nix-build, Nix puts a sym link into your project directory called result that points to the location in the store that holds the build results of the expression. We can use result to get easy access to our new binary.

snap-app

TADA! We are now ready to actually follow the janrain tutorial. This installment is long enough already, so I’ll save exploring the tutorial itself for next time.

Adding Nix Expressions for New Packages

Posted on by
This entry is part 5 of 6 in the series Setting Up Haskell Development on NixOS Linux from Scratch

The nixpkgs repository does a good job of including many Haskell packages already. Indeed, it almost seems like the lazy, functional Nix-based NixOS might have a soft spot for Haskell :). Much of hackage is available with just a few keystrokes, relieving cabal of its package-managing tasks.

However, nixpkgs does not include all of hackage, and it’s actually not hard run into missing packages during development. For instance, not long ago I wanted to build a toy snap app as a first project in NixOS. While snap is available, snap-web-routes is not.

We’ll explore adding hackage packages to your local nixpkgs repo (and submitting them upstream) in this installment. In fact, we’re going to add snap-web-routes to nixpkgs.

Preparing git

If you followed my earlier installment in this series, you should have a local clone of the nixpkgs repository. Since we plan to submit a pull request with our changes, we need to take a few steps to prepare git. If you are pretty seasoned with git, you can probably skip this section.

Add and Sync Upstream

We need to add the primary NixOS/nixpkgs github repository to our remotes. This will enable us to pull in updates to the nix expressions easily.

#!bash
git remote add upstream https://github.com/NixOS/nixpkgs.git
git fetch upstream
git merge upstream/master

The last two lines are what we’ll execute whenever we want to pull in updates for our custom nix-search and nix-install calls.

Make a New Branch

Next we need to create a new branch for adding our new expression. This will isolate our updates when we submit a pull request; otherwise, any additional, unrelated updates we make to our local repository and push to origin will get added to the pull request! I learned this the hard way.

Make a new branch and check it out using checkout and -b.

#!bash
git checkout -b add-haskell-snap-web-routes

Adding Expressions

Nixpkgs Expression Hierarchy

The nix expressions for all the available package are included in our local nixpkgs repository. Each nix expression is specified using a path. If no file name is provided in the path, then the file name of default.nix is assumed. For example, nixpkgs/ has a default.nix, which has the following line

#!nix
import ./pkgs/top-level-all-packages.nix

The default.nix expression imports all the expressions listed in pkgs/top-level/all-packages.nix. This expression, in turn, imports expressions from other locations in the nixpkgs repository. The one we are interested in is pkgs/top-level/haskell-packages.nix. Here’s just one excerpt from the haskell-packages.nix file.

#!nix
snap = callPackage ../development/libraries/haskell/snap/snap.nix{}

So, when we invoke nix-install haskellPackages.snap, Nix calls the expression located at in the ../development/libraries/haskell/snap/snap.nix file. Thus, to add a new packages to our repo, we need to

  1. Write a nix expression file and put it in a logical spot in the nixpkgs repository.
  2. Put a line in haskell-packages.nix we can use to call the expression file.

A Nix Expression for a Cabal Package

To add snap-web-routes to the hierarchy, we need to write a nix expression for it. What does a nix expression look like? Here’s the one for snap given in the nix file mentioned above.

#!nix
{ cabal, aeson, attoparsec, cereal, clientsession, comonad
, configurator, directoryTree, dlist, errors, filepath, hashable
, heist, lens, logict, MonadCatchIOTransformers, mtl, mwcRandom
, pwstoreFast, regexPosix, snapCore, snapServer, stm, syb, text
, time, transformers, unorderedContainers, vector, vectorAlgorithms
, xmlhtml
}:

cabal.mkDerivation (self: {
  pname = "snap";
  version = "0.13.2.7";
  sha256 = "1vw8c48rb1clahm1yw951si9dv9mk0gfldxvk3jd7rvsfzg97s4z";
  isLibrary = true;
  isExecutable = true;
  buildDepends = [
    aeson attoparsec cereal clientsession comonad configurator
    directoryTree dlist errors filepath hashable heist lens logict
    MonadCatchIOTransformers mtl mwcRandom pwstoreFast regexPosix
    snapCore snapServer stm syb text time transformers
    unorderedContainers vector vectorAlgorithms xmlhtml
  ];
  jailbreak = true;
  patchPhase = ''
    sed -i -e 's|lens .*< 4.2|lens|' snap.cabal
  '';
  meta = {
    homepage = "https://snapframework.com/";
    description = "Top-level package for the Snap Web Framework";
    license = self.stdenv.lib.licenses.bsd3;
    platforms = self.ghc.meta.platforms;
  };
})

Whoa, that’s a lot to take in. Let’s go through the main points.

Dependencies List

The big list at the top are the dependencies for the nix expression. When Nix runs this expression, it first ensure that all dependencies are available and placed on the path. Anything not in the dependencies list is removed from the path. If a necessary dependency is not specified in this list, Nix will refuse to run the expression.

One thing to note about these dependencies: they look like Haskell package names, but technically they are nix expressions in the same namespace as the snap.nix expression. By convention, nix expressions for Hackage packages are named slightly differently.

Cabal.mkDerivation

The call to cabal.mkDerivation is a function call defined elsewhere in the expression hierarchy, and each item specified within the curly braces is a named input argument to the function. The important items are mostly self-explanatory, but a couple could use some elaboration:

  • sha256 – a hash of the source code used to build the package. Nix checks the hash for consistency before building the package.
  • jailbreak – strips out the dependency version bounds from the cabal file before building.

Generating a Nix Expression with cabal2nix

Specifying a nix expression like the one for snap from scratch would be a huge PITA. Thankfully, NixOS has a sweet utility function called cabal2nix that essentially handles everything for us.

#!bash
nix-install haskellPackages.cabal2nix

First, make sure your hackages list is up to date.

#!bash
cabal update

Next, call cabal2nix and specify the hackage package name.

#!bash
[dan@nixos:~/Code/nixpkgs]$ cabal2nix cabal://snap-web-routes
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0  7161    0     0  12992      0 --:--:-- --:--:-- --:--:-- 12992
path is ‘/nix/store/wyilc14fjal3mbhw0269qsr5r84c5iva-snap-web-routes-0.5.0.0.tar.gz’
{ cabal, heist, mtl, snap, snapCore, text, webRoutes, xmlhtml }:

cabal.mkDerivation (self: {
  pname = "snap-web-routes";
  version = "0.5.0.0";
  sha256 = "1ml0b759k2n9bd2x4akz4dfyk8ywnpgrdlcymng4vhjxbzngnniv";
  buildDepends = [ heist mtl snap snapCore text webRoutes xmlhtml ];
  meta = {
    homepage = "https://github.com/lukerandall/snap-web-routes";
    description = "Type safe URLs for Snap";
    license = self.stdenv.lib.licenses.bsd3;
    platforms = self.ghc.meta.platforms;
  };
})

Boom. We get a complete nix expression without doing any work. Now we just need to put it in the right spot in the hierarchy. The rest of the hackage packages are in pkgs/development/libraries/haskell, so we’ll follow suit.

#!bash
[dan@nixos:~/Code/nixpkgs]$ mkdir pkgs/development/libraries/haskell/snap-web-routes

[dan@nixos:~/Code/nixpkgs]$ cabal2nix cabal://snap-web-routes > pkgs/development/libraries/haskell/snap-web-routes/default.nix

Now we can check that our expression actually runs. The nix-install function we added isn’t smart enough to handle --dry-run, so we’ll use the nix-env command directly.

#!bash
[dan@nixos:~/Code/nixpkgs]$ nix-env -iA haskellPackages.snapWebRoutes --dry-run
installing `haskell-snap-web-routes-ghc7.6.3-0.5.0.0'
these derivations will be built:
  /nix/store/f42ybqmcmq44nr73l6ap90l5wnm3s4kq-haskell-snap-web-routes-ghc7.6.3-0.5.0.0.drv
these paths will be fetched (21.88 MiB download, 416.87 MiB unpacked):
  /nix/store/0kw75f2qqx19vznsckw41wcp8zplwnl7-haskell-errors-ghc7.6.3-1.4.7
  /nix/store/14rjz7iaa9a1q8mlg00pmdhwwn7ypd4x-haskell-distributive-ghc7.6.3-0.4.4
  /nix/store/1jnipx1swkivg1ci0y7szdljklaj9cx1-haskell-skein-ghc7.6.3-1.0.9
...

So snap-web-routes will be built, and all of its dependencies will be fetched from pre-built binaries. Sweet! If the dry-run looks good, you can run the command again without the --dry-run option or use nix-install like we have previously. I omit that call here.

Note that this will install the package to your user environment. Remember: installing a package means making it active in the user environment. If you don’t want to install it, you can just build it with nix-build, which will put it in your store but it won’t be installed for the user.

#!bash
nix-build -A haskellPackages.snapWebRoutes

Make the Pull Request

Again, if you’re already adept at git and github, you can skip this section. We’re going to push our changes up to github and make a pull request. The process is largely automatic! Add the new .nix file to the git repo, commit the changes, and then push the branch to github.

#!bash
git add pkgs/development/libraries/haskell/snap-web-routes/default.nix
git commit -a -m "Added snapWebRoutes to haskellPackages."
git push origin add-haskell-snap-web-routes

Go to your github repo and you should see a prompt asking if you want to create a pull request; something like NixOS:master -> fluffynukeit:add-haskell-snap-web-routes. Follow the prompts and you’re done!

That’s it for this installment! New installments to come if/when I encounter new problems to write about!