Category Archives: Languages

Installing NixOS

Posted on by
This entry is part 2 of 6 in the series Setting Up Haskell Development on NixOS Linux from Scratch

In the last post of this series, we set up a VM to run NixOS using VirtualBox.  Now we actually need to go ahead and install NixOS.  We’ll be installing it into our VM, but if you are installing it into an actual machine, most of the instructions here will work for you.

This post is based on this article on the NixOS wiki, but there are some differences because I chose to configure NixOS differently.

Installing NixOS

  1. Go to the NixOS download page and download a release of NixOS.  At the time of this writing, the newest release is 14.04, and I chose the Graphical Live CD, 64-bit Intel/AMD version, which is recommended for most users.
  2. Open up the VirtualBox VM Manager and select your NixOS VM.  Then choose Settings > Storage.  Click the CD icon in the storage tree.  Then, on the right side under Attributes, click the CD icon to choose a virtual CD/DVD file.  Select the nixos ISO file you downloaded in step 1.NixOS Boot
  3. Click OK, then start.  You should see the following menu.  Choose the first option, NixOS installer.  It is chosen for you automatically after about 10 seconds.GRUB Installer
  4. After the NixOS installer boots up, you’ll be at a terminal login prompt that tells you to login as root.  Enter root for your login name and hit enter.
  5. So far, we haven’t actually installed anything.  Before we can do that, we need to initialize the file system on our virtual machine so it can read and write files to our virtual disk we set up in the previous post.  The first thing to do is create a partition on our virtual disk.  Type 
    fdisk /dev/sda

    to run the disk partitioning program, which has a simple but completely non-intuitive interface.  The commands I entered, in order, are n (for new partition), p (for primary partition), 1 (assign partition number 1), no selection (defaults to 2048 for start of partition), and no selection (again, default).  Finally, I enter w to write the partition to disk.  See below.Parition Disk

  6. Next, we need to make a file system onto our new partition.  Enter 
    mkfs.ext4 -j -L nixos /dev/sda1

    This command will make a new file system with the ext4 format into partition 1 of sda, enable journaling, and give our new filesystem the label nixos.

  7. Now that we created the filesystem, we need to mount it.  Mounting it is like connecting different filesystems together, but in this case we need to connect our filesystem to the OS.  Mount the filesystem with 
    mount LABEL=nixos /mnt

That’s the last step in setting up the disk and file system.  Next, we’ll configure NixOS itself.

Configuring NixOS

If you just want to get NixOS set up as fast as possible, skip to the instructions below.  Before I get to that, however, I’ll briefly describe the Nix philosophy and implementation from a few different levels.  These are just my impressions and understanding from using NixOS for a few days, but I have found this information to be crucial in order to do anything useful.

NixOS is a Linux distribution built around the Nix package manager, which is available for installation on other OSes if you want, but is the only package manager for NixOS.  Every piece of software installed on the entire system is installed through Nix.

So what makes Nix useful?  In most other package managers, new versions of software replace older versions.  If some component of the system depends on that software but is incompatible with the new version, then it will break after the upgrade.  Extend this problem to the huge graph of intermingling dependencies among all the software on a user’s computer and you get what’s called dependency hell.  Other Linux distributions try to prevent dependency hell by keeping official lists of software packages that are expected to be compatible, installing packages from this list, and occasionally updating the list with new versions.

Nix, on the other hand, installs new version of software side-by-side with their older versions, and older versions are deleted only when they are no longer required.  Nix achieves this through a combination of two mechanisms: a sophisticated system of sym links that keeps track of which version of packages are “active” at any time and a build system that exhaustively describes every input into each individual package.

The second part is the part we need to understand for our next steps.  Installing new software requires writing a Nix expression that completely describes all the prerequisites and dependencies for the software to be installed.  These dependencies can themselves be Nix expressions defined elsewhere.  When we install new packages, what we are doing is running a Nix expression for that package.  Nix first recursively ensures all build dependencies are available, either by checking that they are already built or by running Nix expressions to build them.  Then, it builds and installs our new package.

This is exactly the process we’ll use to install NixOS itself.

  1. First, we need an expression that will define the components we want to build and install.  We can make one and configure it using 
    nixos-generate-config --root /mnt 
nano /mnt/etc/nixos/configuration.nix

    This will open the configuration.nix expression in the nano editor.

  2. The configuration looks something like the one below.  Use the arrow keys to navigate through the file.  Lines marked with a # are comments, and several important configuration lines are commented out by default.  Use nano to move the cursor around and delete the # to enable the features you want.  In the picture, I have deleted the #s for boot.loader.grub, networking.hostName, and networking.wireless.enable.  I also uncommented all the lines that begin with services.xserver to enable a GUI.  I also uncommented the lines at the bottom of the file that add a new user and modified them as shown.  When you are done, use Ctrl-O to save the changes and Ctrl-X to exit.Nix configNix User Config
  3. With the configuration.nix nix expression defined, we are ready to install NixOS! Run 
    nixos-install

    to start the installation.  If there is a problem with the filesystem and the installation refuses to start, you can redo the steps starting with the fdisk step, but use d (for delete) as the first command to delete the primary partition and start fresh.

  4. After installation completes, shut down the virtual machine, and in the VirtualBox Manager go to Settings > Storage, click on the CD on the right hand side and choose “Remove disc from virtual drive.”  This removes the NixOS installer from the boot menu so we don’t boot into it by default.  Then start the VM again.
  5. When the VM boots up, you should see a graphical login window.  We can’t log in yet because we need to change some passwords first.  To do this, we need to use the command line again.  Access the console login using the Power drop down menu.Console Login
  6. Log in as root, which still has no password.  We will change that right now with 
    passwd

    Enter a new root password for your system.  If you created any users, you can set their passwords also with, for example,

    passwd dan
  7. reboot one final time and verify you can log in successfully.

That’s it! whew Next in the series, we’ll tackle how to install essential software.

Installing VirtualBox (for NixOS)

Posted on by
This entry is part 1 of 6 in the series Setting Up Haskell Development on NixOS Linux from Scratch

Welcome to the first installment of my series about setting up Haskell development on NixOS Linux from scratch!

I consider myself somewhere in the beginning-intermediate stages of Haskell skill, having created a useful application for Windows as learning exercise.  However, to really take my Haskell skill to the next level, I needed to migrate to the de facto standard Haskell environment – Linux.  Ollie Charles, a Haskell reddit member who writes the wonderful 24 Days of Hackage tutorial series on his site, has adopted NixOS as his Linux distro of choice and wrote a blog post describing how to get started.  I decided to trust his choice and try it out for myself.

As I dove deeper into the rabbit hole of getting my environment set up properly, I found myself getting frustrated with having to search all over the net for tutorials, walkthroughs, and installation help just to get everything set up.  Indeed, Ollie’s post does a good job giving the flavor of Haskell development on NixOS, but it’s not comprehensive enough to provide the whole story.  The goal of this series is to provide that whole story as I discover it myself.  If something in these posts is unclear or flat out wrong, please let me know!

Ok, with the introduction out of the way, let’s jump right in.

Installing VirtualBox

VirtualBox is a free program from Oracle that allows you to run a virtual machine on a host computer.  In this case, we’ll use a virtual machine to run NixOS and host the VM on Windows.

  1. Download VirtualBox installer from here.  At the time of this writing, the latest version is 4.3.12.  Since we are hosting the VM on a Windows computer, we want VirtualBox 4.3.12 for Windows hosts.
  2. Run the installer.  You’ll see something like the screen below.  I accepted all the defaults except I don’t like shortcuts on my desktop, so I disabled that (on the next screen).  Also, I got a warning that I would be temporarily disconnected from my network as part of the installation process.Virtual Box Installer
  3. Wait for installation to complete.  If you are prompted to install devices, go ahead and install them.  Installation took 10-20 seconds for me.

Create a virtual machine to run NixOS

Now that we have VirtualBox installed, we need to create a virtual machine that will run our NixOS distribution.

  1. Start your newly installed VirtualBox software.  When you start it up, you’ll be at the VirtualBox Manager window.  From this window, you can create and manage multiple virtual machines that all run inside the host computer.  We only need to create one, though.  Click the blue New button.
  2. Now we’re at the Create Virtual Machine Wizard, which looks something like the window below.  Your selections will be similar to mine but might need to be different based on your computer architecture.  I have a 64 bit machine, so I chose Other Linux (64-bit).  You can also use whatever Name you like.  I happen to know I will be using NixOS 14.04.Create VM
  3. Next you will choose the amount of RAM for the VM.  This is the amount of RAM your VM reserves for its own use while you are running it.  My machine has 8 GB total, so I allocated 4 GB (4096 MB) to my VM because I never plan to run it at the same time as another RAM-hungry process.VM Ram Size
  4. At the next screen, you are prompted to create a virtual hard drive.  Choose to create a new drive now and choose the VDI format on the next screen.  After that, choose “Dynamically Allocated,” which allows your VM to take up hard drive space as it needs it instead of reserving it all up front.  Lastly, choose the size of your new virtual drive.  The default setting for me is 8 GB, which sounds sufficient for small development box.  Besides, if we run low on space, we can increase the size later.VM Hard Drive Size
  5. After that, your VM is created!  You can now click on the orange Settings gear to explore different ways to configure your VM.  I plan to run with the defaults until I decide I need more processing power or otherwise need to change the settings. The only thing I changed was under General ? Advanced where I changed Shared Clipboard and Drag’n’Drop to both be bidirectional.

And that’s it! Don’t bother starting the VM yet, though.  There’s no point until we install the NixOS operating system onto it.  We’ll cover that in Part 2.

 

 

FNIStash r1.4 released!

Posted on by

It’s been a long while, but a new version of FNIStash is available. r1.4 adds only one feature, but it will be really useful for some people. In this release, you can optionally configure where the fnistash.db file is located on your system. You can even point it to a network drive! Now you and a friend can share the same FNIStash database, but be careful: only one of you should run FNIStash at a time.

This release is a change I made for a user request quite some time ago but never got around to releasing. It has worked for her without any problem that I know of.

So I looked at the download logs for FNIStash releases, and there are now several hundred users!  Not bad.

The last 6 months has been a tumultuous time for me.  My wife got a new job, I left my old job and got a new one, and I moved to the Seattle area.  The process has been MUCH longer than I would have liked or anticipated, but today officially marks the end of it because my wife is finally joining me in Washington.

FNIStash r1.3 released!

Posted on by

It’s been a pretty wild 1-2 months since I released 1.0 of FNIStash, both for me personally and for the program itself.  I’m glad that it has been very warmly received and that so many people are giving me good feedback.  It has gone through a few subsequent releases to fix bugs and add new features, the biggest probably being the ability to delete items from the Registry.  It has been downloaded over 150 times, which is pretty good I think considering I don’t publicize it much.

Other things in my life include developing a website with my brother (which, sadly, we lost our momentum on, but I hope we can get it back in the future), illness, my mother-in-law returning to China after a bureaucratic nightmare, and a shakeup at work.  So my attention to this blog has waned, but I expect this kind of transitional period in my life to get wrapped up in short order.

FNIStash r1.0 released!

Posted on by

It’s been a long time coming, but I have made the first release of FNIStash!  I consider this release experimental until all the bugs are worked out.  So far I have already received some good feedback and a couple of bug/compatibility reports.  Follow the links below for more.

FNIStash homepage, with video tutorial

Announcement of FNIStash on Runic Games Forum

Announcement of FNIStash on r/Torchlight subreddit

It was a long, educational experience writing this program.  I think I got a good feel for Haskell and hope my next project with the language focuses more on good habits and style instead of focusing almost solely on making a functional product.  I’d like to eventually write up a post-mortem of my experience of building a non-trivial application in Haskell as a 100% FP and Haskell newb, so look for that in the future if you’re interested.

Improving memory performance in Haskell programs

Posted on by

I ran into a big problem recently in FNIStash – I realized that it was using way more memory than I thought it should.  What followed was a couple weeks of arduous investigation and learning about Haskell.  If you find your program has a memory leak that is not due to thunks, these tips might help you out.  Note that this all applies to the de facto standard compiler, GHC.

Understanding memory usage

The first step to figuring out why your program is using a lot of memory is to measure it.  You can enable profiling by compiling your program with the -prof and -rtsopts compilation flags.  This enables profiling and allows you to control it from the RTS options passed from the command line when you run the program.  For example,

MyProgram.exe +RTS -hy

will run the program and categorize memory usage by type.  There are lots of categorization options available, such as by module or function.

When your program terminates, you’ll get a .hp file in the same directory as the executable.  You can convert it to a viewable PS or PDF file using

hp2ps -e8in -c MyProgram.hp 
ps2pdf MyProgram.ps

The result is something like this.

Memory profile of FNIStash categorize by type.

Memory profile of FNIStash categorize by type.

Here we can see that most of the memory is used up by ARR_WORDS, which is a backing type for ByteString and Text, among other things.  Since FNIStash reads and parses an archive of binary files, this makes sense.  However, the magnitude of the plot maxes out at around 25 MB.  Why, then, does Windows report over 70 MB of memory used?

Memory footprint of FNIStash

The discrepancy is due a couple of factors:

  1. First, the profiling doesn’t come for free.  Compiling with the -prof option necessitates more memory just for reporting purposes.  This requires approximately 30% more memory than usual.  This overhead isn’t reported on the plot, but the OS sees it.
  2. The reported memory usage is only “live memory.”  It doesn’t include old data that has not been collected by the garbage collector yet.  In fact, in very loose terms, the default GHC GC waits until live data grows to the size of old data before collecting it.  This means your program might use twice as much memory as needed.  Note that when the garbage collector does run, it temporarily needs more space to reorganize the live data, so if a large amount of data is being collected at once, you can see memory usage (as reported by the OS) spike up.
  3. FNIStash, in particular, has some kind of memory leak.  Right after start up, the OS memory usage was around 66 MB, but crept up to 70 MB after some minor GUI interaction.  It’s actually unclear to me whether the leaked memory is reported in the plot.

So if we discount the leaked memory, then we have 66 MB / 1.3 / 2 = 25 MB, which is right in line with what the graph reports.  Turning off profiling saves an easy 30% of OS memory usage, but what if you want to save more?  How can the memory leak be fixed?  Are you stuck with using twice as much memory as necessary?

Fixing the leak

Fixing the memory leak was a Sisyphean process of changing data types, adding strictness, and other futile efforts until I found the magic bullet: the threaded run time system.  By default, Haskell uses a single-threaded RTS (meaning 1 OS thread) that can nonetheless provide concurrency.  Indeed, FNIStash uses multiple threads to run the frontend and backend separately.  The -threaded flag enables the multi-threaded RTS (multiple OS threads), and using it completely obliterated the terrible memory creep I was seeing in the Task Manager.  I don’t know why this is.  My suspicion is that the idle-time GC that comes with -threaded more aggressively cleans up.  Given the number of cores on modern machines, I plan to make this my standard in the future.

Tuning the garbage collector

GHC enables the user to tune the garbage collector from the command line using RTS arguments.  Here are the ones that I tested out.  To use them, your program must be compiled with the -rtsopts flag.  Additionally, and this is totally anecdotal on my part, it seems that having -prof enabled at the same time prevents any of these options from having an effect, so if you notice this you might want to compile without -prof.

  • -c : This option changes the GC algorithm to use a slower compacting strategy instead of a copying strategy.  What I think this mean is that instead of copying bytes around the heap when running GC, data gets compacted near where it already is.  The end result is less heap usage.  In my case, using this option yielded savings of around 10%.
  • -Ffactor : Here “factor” is a value.  By default, GHC uses a value of 2.  This is the growth factor allowed before GC is run.  If you make this smaller, GC will be run more often.  However, this didn’t seem to help me out much.
  • -Gx : If x = 1, this enables the single generation collector.  I used this option without any others, and it actually increased my memory usage considerably and made my program much slower.  I also tried G3, but that used a lot more memory as well.

In the end, I decided that killing the leak with the threaded RTS was enough for me.  I don’t need extra memory savings by using -c for the moment.  So I went from 70+ MB with a leak down to 48 MB with no leak by using -threaded and removing -prof.  The real benefit, however, was learning more about GHC’s runtime system.

Emulation of TL2 Shared Stash

Posted on by

A lot of cool news to report on FNIStash!

After successfully pulling png icons out of the TL2 game data, the next step was to get a GUI going.  As I’ve written about previously, GUIs on Haskell (especially windows) are a PITA.  However, after doing a clean reinstall of Haskell Platform, I was able to successfully install Ji and build the example executables.

What is Ji?

The GUI landscape on Haskell is pretty bad.  There are lots of frameworks out there, but actually installing them (much less using them) is often a big pain.  There really hasn’t been much effort lately to make these frameworks more user friendly.

Where does Haskell get a lot of attention?  The Web.  Ji is a package that takes advantage of this web attention by turning your web browser into desktop GUI.  You control the browser GUI from your Haskell application with commands to add elements, handle events, etc.  Ji achieves this by running a Javascript library on your page that transparently communicates with the Haskell server using JSON to execute commands and send event notifications.  The low latency of a desktop connection to itself makes this feasible, but technically Ji could be used for web apps that have a sufficiently fast connection (probably not a good idea, though).  It’s all automatic and requires very little set up.

For FNIStash, I needed to expand on the Ji package by enabling assignment of element IDs and lookups thereof.  I found that most of the infrastructure for this was already there but unused, which made adding the functionality pretty easy once I figured out how the package worked.  The updates went into Ji earlier this morning, but  Ji’s creator, Chris Done, has other things on his plate and wanted to unload responsibility for the package.  Heinrich Apfelmus, of reactive-banana FRP fame, volunteered to take the project over with my assistance.  That transition is still in the works.

Ok, so FNIStash…

I recently discovered how TL2 stores the location information for items.  There is a hierarchical system of containers, slots, and indices, each of which has an ID to keep track of it all.  From a raw item binary, I now know where that item is among TL2’s various item containers.  The result?

Image of working shared stash emulation

A working shared stash display!

This is a working shared stash display, albeit unpolished.  Each item is inserted into the emulated stash in its same position just like the game.  The selection tabs at the top work, so clicking on a different tab changes the class of items that are displayed.  Ji handles this all over the wire, although I admit is was a little tricky getting the click event handler written.

The error at the top indicates that there are some items in this shared stash file that I am not parsing correctly.  It’s true that there are cases my parsing logic doesn’t currently handle, mainly because I don’t understand the file format entirely.  For the time being, handling most cases is sufficient for moving development forward.

It’s actually starting to look like a real, genuine desktop app.  I’m excited!

A comparison of Haskell vs F#

Posted on by

In my last post, I wrote about my frustrations as a Haskell user on Windows.  It led to a good discussion on Reddit about a number of points I raised, as well as some good comments on this site.  I received more than one recommendation to try out F# as an alternative to Haskell.  I’ve been casually exploring it for a few weeks now, and I have decided not to pursue it.  My impressions are below.  Keep in mind that I am coming to F# as a barely experienced Haskell user, but a Haskell user nonetheless.  My impressions are also restricted to F# in Windows, as I did not investigate Mono or other Unix tools.

F# – The Good

Full citizen status – Most immediately noticeable when coming to F# from Haskell is that Windows users come first.  You do not need to install any Unixy tool chains to build what you want.  In fact, almost everything you could possibly want is downloadable straight from Microsoft’s enormous and comprehensive .Net framework.  Everything just works.

Documentation – Being part of the .Net framework, F# has a lot of corporate resources behind it at MS.  This includes in depth documentation about nearly every .Net component.  What is more, the .Net community is enormous, so finding examples online for pretty much anything is much easier than it is for Haskell.

Advanced tooling – One of the things Haskell suffers from, in my opinion, is lack of advanced development tools, including a standard and full-featured IDE.  In F#, you get Visual Studio, a fully featured IDE that has been tested by thousands, if not millions.  Haskell has no equivalent.  Leksah is decent, but it is nowhere near as capable as VS.

For example, in F# on VS you can scroll over (almost) any value or function and see the inferred type signature for it.  In Haskell, if I have a tough type error, I usually have to start explicitly defining the types of values and functions to narrow down the scope of my mistake.  In VS, I can just scroll over the item in question and immediately get hints as to where I went wrong.  Unfortunately, as far as I know this only works for named functions and not functions defined as symbols (operators).

Sane strings – Haskell’s default implementation of strings is a linked list of characters.  While providing some advantages, in most cases it’s plain inefficient and unusable.  GHC has some extensions to help fix this, such as the OverloadedStrings extension, and there are libraries to help as well, such as the text library.  But different libraries choose to handle strings in different ways, so you often find yourself calling a number of conversion operations just to glue all the right types together.  F#, on the other hand, has one standard string implementation that everyone uses.

Standards – Somewhat similar to the previous point, the Haskell ecosystem sometimes feels like the wild west, with different projects experimenting with types and strategies.  Frequently, this can lead to ambiguity in how a newbie should approach a particular problem.  In F#, there is usually a standard, recommended way to do things – the .Net way.

Computation expressions – This is a feature in F# similar to monads that is in some ways less powerful (see below) and in some ways more powerful.  Like do notation, CE’s are syntactic sugar for monad-like operations.  The familiar Bind and Return are present, like Haskell, but F# allows additional desugaring of loop expressions, the ability to implement lazy evaluation, and more.

No lazy evaluation (unless you want it) – Arguably a con overall, but my point here is that reasoning about performance and space usage is easier in F# due to strict evaluation semantics.  The profiling utilities built into VS make this even easier still.

F# – The Bad

Verbose and odd syntax – A number of aspects of F# rub me the wrong way.

First, every declaration requires a let keyword.  Let let let let let.  Haskell in a lot of ways is much more concise because several values can be declared using one let.

Secondly (and this drives me nuts), F# function and type declarations must be made before the function is used.  In Haskell, I often use a few helper functions that are defined down near the bottom of the file, but in F# those helpers must be defined near the top of else they can’t be used.  Having such an antiquated restriction as this in a language a modern as F# boggles my mind.

Thirdly, the declaration of types and their constructors is sometimes not consistent.  For instance, a constructed tuple in F# is (a,b) (like Haskell), but the type of this tuple is c * d.  It’s not a hard thing to get over, but it does accentuate how self-consistent Haskell is.

No type classes! – I put an exclamation point on this one because it might be the single most important reason F# left a bad taste in my mouth.  I did not realize the power of type classes until I tried to do functional programming without them.  Take for example the monad type class.  If a type is an instance of the monad type class, it is a given that it will work with replicateM, sequence, and a number of other functions that work on monads.  If you have a new type, just declare a monad instance for it and sequence will work with it as well.

Trying to do this in F# is an exercise in frustration.  There are ways to simulate type classes, but you have to do all kinds of inlining and type gymnastics to make the .Net CLR happy.  Since it’s not readily supported by the language, there is no standard library of monad functions, and in a lot of situations you’ll have to reimplement these extraordinarily useful utilities yourself.  This can lead to a good deal of code duplication.  Just check out the FSharpx monad library code for examples.

Dependence on .Net – .Net is a comprehensive, full-featured framework, but it is huge and designed for OO languages.  This results in classes like Stream, StreamReader, BinaryReader, TextReader, etc., which are all slight variations on each other.  F# uses these same classes.  Digging through the docs to find what you need can be challenge, but less so if you’re already familiar with .Net.  In Haskell though, almost all of these can be replaced by mapping to/from a plain old byte string.  Why does OO have so much seemingly unnecessary complexity to simply deserialize a type?

“Functional first” – F# is often described as a “functional first” language, meaning it supports both functional and OO programming paradigms.  For some this is a plus (right tool for the right job and all that), but to me it seems like F# suffers from split personality disorder.  Some of the F# libraries feel like little more than functional wrappers around an OO core.  For a newbie, it’s not entire clear if these are merely measures of convenience or of necessity.

Back to Haskell

I doubt I was able to completely commit my thoughts on F# to this page, but this is at least a good start.  I’ve decided to continue to wrestle with Haskell on Windows because even though the ecosystem is a house of cards, actually coding in Haskell is much more enjoyable than F#.  I’ve started over with Haskell Platform to hopefully resolve as many compatibility issues as i can.

Reflections After a Hard Day in Haskell GUI Land

Posted on by

My progress on my current project (FNIStash) is unsteady, but yesterday I decided the time was right to start putting together a rudimentary GUI as both a learning exercise and as a boost of motivation for moving the program along.  After working on getting a basic toy GUI for an entire day, I have absolutely nothing to show for it.  It was truly the most disheartening and disappointing interaction with the Haskell ecosystem that I have had so far.  Here are some of my impressions.

As a Windows User

Trying to navigate the Haskell ecosystem as a Win 7 user is like making a deposit at a bank in loose change.  Some banks will accommodate you and others will turn you away, but in either case you’ll get the feeling that the way you are doing things is fundamentally looked down upon.  In cases where you are accommodated, you usually have to jump through many hoops, such as installing specialized build environments like MINGW32 or cygwin (which themselves might require several different installers) and manually building cross-platform libraries.  After a lot of work, you’ll find you’re ready to install the package you wanted to install all along, only to have it break for who-knows-why.  A lot of libraries are just not as easy as “cabal install haskell-gui-support”

Cabal

Cabal is a frustratingly constraining tool.  Far too frequently I encountered packages that, when trying to install, would say installing this package will break a dozen others.  If not that, then I often would be notified that the dependencies could not be resolved.  If you decide that you don’t care that your other packages break and go ahead with the install, it’s a tossup as to whether or not the install will be successful, but oops, you just broke a dozen of your old packages already, so if it fails you have some cleaning up to do.  I’ve become pretty skilled at clearing out my cabal and ghc databases and invoking cabal install world to just bring my packages back to a self-consistent state after trying and failing to install a new package to explore.

For packages that leverage a cross platform library, this is frustraing but understandable.  Often this house of cards is built by taking C or C++ source meant for a Unix like build environment, installing it through MINGw32 (as the special Windows installation instructions tell you to do), then installing a package from cabal that brings in the Haskell wrappers.  It’s not pretty, but it’s kind of expected when pulling in functionality across platforms and across languages.

What I don’t understand is why cabal should ever have a problem with some packages breaking others or dependencies not being resolved.  Cabal knows what package I want to install, it knows what versions of its dependencies it needs, and those dependencies are freely available from hackage (as are their own dependencies).  You would think that this is a simple scenario with a straight forward solution – recurse and build all versions of everything as needed.  But for some reason, cabal does not allow side-by-side installation of different versions of packages.  If you have a package that requires a specific version of a dependency, you are effectively prevented from using any package that requires a different version.  The utility cabal-dev is available, which allows the set of packages to be sandboxed on a per project basis, but that is small consolation if you (like me) need to get out of hell on a single project.

Side-by-side installation of multiple versions is a known and demonstrated solution to dependency hell.  I alternate between blaming the community for not solving this problem and blaming myself for not understanding why it is so difficult.  Solutions like Haskell Platform seem to me to be treating the symptom and not the disease.

Haskell Community

The Haskell community is widely regarded as newb-friendly.  If you need newb help, the haskell IRC channel is just a click away, and in cases where I didn’t get help from the IRC channel or haskell cafe, I’ve simply selected a knowledgable member of the Haskell subreddit and messaged them directly.  No one has ever called me a newb or made me feel dumb for asking questions, even when the answer was staring me in the face in the docs and I just didn’t notice.  It’s a great community.

What has happened (albeit occasionally) is that I might get no response at all, and as my questions become more advanced and specific, the likelihood that my queries get lost in the void increases.  This is no different from any other language community.  However, the Haskell community is tiny compared to, say, Python, so it’s pretty easy to exhaust all avenues of help.  If you’re in the Haskell Windows community, it’s even smaller.

Desperation in Finding a GUI Package

In my search to find a viable GUI package, the only one that I had absolutely any success with was HGamer3d, and it’s probably not a coincidence that it is designed to work on Windows.  It was the only package that I was able to install successfully (without any headaches either), and it also had several example sources that worked effortlessly.  The problem is that is has next to nothing for documentation.  I tried modifying an example slightly only to have the executable fail on startup for unknown reasons.  Maybe I will revisit it in the near future, but I would prefer to not have to learn Ogre to decipher the HGamer3d source code.

What I wanted at the start of all of this was a GUI library or framework that supported DDS texture files.  By the end, I was willing to settle for anything that would successfully install and would display an image in any format.  So why not the web frameworks?  Yesod can’t resolve dependencies, happstack fails to load the network package (for which I found a work around online that, alas, doesn’t work), and snap fails to build the lens package due to a segfault in the generated code.  I’m sure any of these would work if I had the “right” set of packages installed, but this is not a viable solution (see above).  I can’t rip my environment apart every time I need something new.

wxHaskell is the most popular GUI toolset based on this reddit discussion, but I forgot to sacrifice a goat as step 0 of the installation process, so I had very little success with it.

Now

I feel pretty similar to the author of this reddit discussion, except I had even less success.  GLUT, GLFW, elm, reactive this, FRP that.  I feel as if I have looked at everything and found success in nothing.  Others work in Haskell GUIs all the time, so I know it is possible and that part of the blame lies on my side of the screen.

This experience has been a completely different kind of difficulty compared to learning the Haskell language.  List comprehension, immutable variables, monads, type classes…these are all well documented concepts with essentially no variability from case to case.  Read up long enough and eventually they click, and you are on your way.  The problem I face now is that I don’t know where to go from here.

String Keys Suck

Posted on by

In my previous post, I mentioned that I needed 95 MB of space memory just to run my simple test of extracting data from the Torchlight 2 PAK file.  I did some investigating to figure out what the heck was going on.  The culprit: using a FilePath (which is just a String) as the key to my map.

Prepping for profiling

To profile in Haskell with GHC, you need to compile your program with the -prof option, and throw in -auto-all to automatically add cost centers to the profile output.  You then execute the program with some additional flag to tell the runtime to collect profiling data.  After that, you can look at the resulting .prof file for nice tabular data, but I prefer graphs.  There are a few annoying steps to this whole process, so I created this batch script to handle most of it for me, which I named hprofile.  It runs the exe, generates the products, and tags the prof and graph with a description.

@echo off
%2.exe %3 %4 %5 %6 %7 %8 %9
hp2ps -e8in -c "%2.hp"
DEL %2.hp
set ID=%~1
set newname=%2(%ID%)
IF EXIST "%newname%.prof" (DEL "%newname%.prof")
RENAME "%2.prof" "%newname%.prof"
IF EXIST "%newname%.ps" (DEL "%newname%.ps")
RENAME "%2.ps" "%newname%.ps"
DEL "%2.aux"
CALL ps2pdf "%newname%.ps"
DEL "%newname%.ps"

Baseline – with String keys

Here’s the graph resulting from

>hprofile "baseline" FNIStash-Debug +RTS -p -hc
Memory usage for String keys in map

Memory usage for String keys in Map

The forText2/pakFileList is the function that generates the keys in the Map.  In this case, the keys are Strings (FilePaths).

Improvement – with Text keys

I changed the type of the key in the map from FilePath to Text.  This actually made a lot of sense since I parsed them out as Text anyway, but chose FilePath before so I could use the path handling utilities in System.FilePath.  The lookup function on the map still takes a FilePath as the key.  Now, however, the FilePath is converted to Text within the lookup function.  Here is the result.

Memory usage for Text keys in Map

Memory usage for Text keys in Map

No more runaway memory usage!  The moral of the story: avoid String.