Emulation of TL2 Shared Stash

A lot of cool news to report on FNIStash!

After successfully pulling png icons out of the TL2 game data, the next step was to get a GUI going.  As I’ve written about previously, GUIs on Haskell (especially windows) are a PITA.  However, after doing a clean reinstall of Haskell Platform, I was able to successfully install Ji and build the example executables.

What is Ji?

The GUI landscape on Haskell is pretty bad.  There are lots of frameworks out there, but actually installing them (much less using them) is often a big pain.  There really hasn’t been much effort lately to make these frameworks more user friendly.

Where does Haskell get a lot of attention?  The Web.  Ji is a package that takes advantage of this web attention by turning your web browser into desktop GUI.  You control the browser GUI from your Haskell application with commands to add elements, handle events, etc.  Ji achieves this by running a Javascript library on your page that transparently communicates with the Haskell server using JSON to execute commands and send event notifications.  The low latency of a desktop connection to itself makes this feasible, but technically Ji could be used for web apps that have a sufficiently fast connection (probably not a good idea, though).  It’s all automatic and requires very little set up.

For FNIStash, I needed to expand on the Ji package by enabling assignment of element IDs and lookups thereof.  I found that most of the infrastructure for this was already there but unused, which made adding the functionality pretty easy once I figured out how the package worked.  The updates went into Ji earlier this morning, but  Ji’s creator, Chris Done, has other things on his plate and wanted to unload responsibility for the package.  Heinrich Apfelmus, of reactive-banana FRP fame, volunteered to take the project over with my assistance.  That transition is still in the works.

Ok, so FNIStash…

I recently discovered how TL2 stores the location information for items.  There is a hierarchical system of containers, slots, and indices, each of which has an ID to keep track of it all.  From a raw item binary, I now know where that item is among TL2’s various item containers.  The result?

Image of working shared stash emulation

A working shared stash display!

This is a working shared stash display, albeit unpolished.  Each item is inserted into the emulated stash in its same position just like the game.  The selection tabs at the top work, so clicking on a different tab changes the class of items that are displayed.  Ji handles this all over the wire, although I admit is was a little tricky getting the click event handler written.

The error at the top indicates that there are some items in this shared stash file that I am not parsing correctly.  It’s true that there are cases my parsing logic doesn’t currently handle, mainly because I don’t understand the file format entirely.  For the time being, handling most cases is sufficient for moving development forward.

It’s actually starting to look like a real, genuine desktop app.  I’m excited!

Success in Extracting TL2 Icons

I recently got back to working on FNIStash in Haskell after some diversion into F#, and I had a great breakthrough in developing yesterday.  I successfully installed the Codec-Image-Devil package of bindings to the DevIL image library, which I then used to read the DDS files that are archived in TL2’s PAK archive.  For each one of these DDS files there also exists and IMAGESET (XML) file that defines the name and location of each icon in the DDS file.  Using this information, I wrote out each icon to disk as a PNG file.  My goal is that eventually these will be hosted through an HTML interface to simulate the TL2 stash.

I was somewhat selective about the icons I wrote to disk, and I still got out nearly 1000 icons.  Here are a few:

A comparison of Haskell vs F#

In my last post, I wrote about my frustrations as a Haskell user on Windows.  It led to a good discussion on Reddit about a number of points I raised, as well as some good comments on this site.  I received more than one recommendation to try out F# as an alternative to Haskell.  I’ve been casually exploring it for a few weeks now, and I have decided not to pursue it.  My impressions are below.  Keep in mind that I am coming to F# as a barely experienced Haskell user, but a Haskell user nonetheless.  My impressions are also restricted to F# in Windows, as I did not investigate Mono or other Unix tools.

F# – The Good

Full citizen status – Most immediately noticeable when coming to F# from Haskell is that Windows users come first.  You do not need to install any Unixy tool chains to build what you want.  In fact, almost everything you could possibly want is downloadable straight from Microsoft’s enormous and comprehensive .Net framework.  Everything just works.

Documentation – Being part of the .Net framework, F# has a lot of corporate resources behind it at MS.  This includes in depth documentation about nearly every .Net component.  What is more, the .Net community is enormous, so finding examples online for pretty much anything is much easier than it is for Haskell.

Advanced tooling – One of the things Haskell suffers from, in my opinion, is lack of advanced development tools, including a standard and full-featured IDE.  In F#, you get Visual Studio, a fully featured IDE that has been tested by thousands, if not millions.  Haskell has no equivalent.  Leksah is decent, but it is nowhere near as capable as VS.

For example, in F# on VS you can scroll over (almost) any value or function and see the inferred type signature for it.  In Haskell, if I have a tough type error, I usually have to start explicitly defining the types of values and functions to narrow down the scope of my mistake.  In VS, I can just scroll over the item in question and immediately get hints as to where I went wrong.  Unfortunately, as far as I know this only works for named functions and not functions defined as symbols (operators).

Sane strings – Haskell’s default implementation of strings is a linked list of characters.  While providing some advantages, in most cases it’s plain inefficient and unusable.  GHC has some extensions to help fix this, such as the OverloadedStrings extension, and there are libraries to help as well, such as the text library.  But different libraries choose to handle strings in different ways, so you often find yourself calling a number of conversion operations just to glue all the right types together.  F#, on the other hand, has one standard string implementation that everyone uses.

Standards – Somewhat similar to the previous point, the Haskell ecosystem sometimes feels like the wild west, with different projects experimenting with types and strategies.  Frequently, this can lead to ambiguity in how a newbie should approach a particular problem.  In F#, there is usually a standard, recommended way to do things – the .Net way.

Computation expressions – This is a feature in F# similar to monads that is in some ways less powerful (see below) and in some ways more powerful.  Like do notation, CE’s are syntactic sugar for monad-like operations.  The familiar Bind and Return are present, like Haskell, but F# allows additional desugaring of loop expressions, the ability to implement lazy evaluation, and more.

No lazy evaluation (unless you want it) – Arguably a con overall, but my point here is that reasoning about performance and space usage is easier in F# due to strict evaluation semantics.  The profiling utilities built into VS make this even easier still.

F# – The Bad

Verbose and odd syntax – A number of aspects of F# rub me the wrong way.

First, every declaration requires a let keyword.  Let let let let let.  Haskell in a lot of ways is much more concise because several values can be declared using one let.

Secondly (and this drives me nuts), F# function and type declarations must be made before the function is used.  In Haskell, I often use a few helper functions that are defined down near the bottom of the file, but in F# those helpers must be defined near the top of else they can’t be used.  Having such an antiquated restriction as this in a language a modern as F# boggles my mind.

Thirdly, the declaration of types and their constructors is sometimes not consistent.  For instance, a constructed tuple in F# is (a,b) (like Haskell), but the type of this tuple is c * d.  It’s not a hard thing to get over, but it does accentuate how self-consistent Haskell is.

No type classes! – I put an exclamation point on this one because it might be the single most important reason F# left a bad taste in my mouth.  I did not realize the power of type classes until I tried to do functional programming without them.  Take for example the monad type class.  If a type is an instance of the monad type class, it is a given that it will work with replicateM, sequence, and a number of other functions that work on monads.  If you have a new type, just declare a monad instance for it and sequence will work with it as well.

Trying to do this in F# is an exercise in frustration.  There are ways to simulate type classes, but you have to do all kinds of inlining and type gymnastics to make the .Net CLR happy.  Since it’s not readily supported by the language, there is no standard library of monad functions, and in a lot of situations you’ll have to reimplement these extraordinarily useful utilities yourself.  This can lead to a good deal of code duplication.  Just check out the FSharpx monad library code for examples.

Dependence on .Net – .Net is a comprehensive, full-featured framework, but it is huge and designed for OO languages.  This results in classes like Stream, StreamReader, BinaryReader, TextReader, etc., which are all slight variations on each other.  F# uses these same classes.  Digging through the docs to find what you need can be challenge, but less so if you’re already familiar with .Net.  In Haskell though, almost all of these can be replaced by mapping to/from a plain old byte string.  Why does OO have so much seemingly unnecessary complexity to simply deserialize a type?

“Functional first” – F# is often described as a “functional first” language, meaning it supports both functional and OO programming paradigms.  For some this is a plus (right tool for the right job and all that), but to me it seems like F# suffers from split personality disorder.  Some of the F# libraries feel like little more than functional wrappers around an OO core.  For a newbie, it’s not entire clear if these are merely measures of convenience or of necessity.

Back to Haskell

I doubt I was able to completely commit my thoughts on F# to this page, but this is at least a good start.  I’ve decided to continue to wrestle with Haskell on Windows because even though the ecosystem is a house of cards, actually coding in Haskell is much more enjoyable than F#.  I’ve started over with Haskell Platform to hopefully resolve as many compatibility issues as i can.

Reflections After a Hard Day in Haskell GUI Land

My progress on my current project (FNIStash) is unsteady, but yesterday I decided the time was right to start putting together a rudimentary GUI as both a learning exercise and as a boost of motivation for moving the program along.  After working on getting a basic toy GUI for an entire day, I have absolutely nothing to show for it.  It was truly the most disheartening and disappointing interaction with the Haskell ecosystem that I have had so far.  Here are some of my impressions.

As a Windows User

Trying to navigate the Haskell ecosystem as a Win 7 user is like making a deposit at a bank in loose change.  Some banks will accommodate you and others will turn you away, but in either case you’ll get the feeling that the way you are doing things is fundamentally looked down upon.  In cases where you are accommodated, you usually have to jump through many hoops, such as installing specialized build environments like MINGW32 or cygwin (which themselves might require several different installers) and manually building cross-platform libraries.  After a lot of work, you’ll find you’re ready to install the package you wanted to install all along, only to have it break for who-knows-why.  A lot of libraries are just not as easy as “cabal install haskell-gui-support”

Cabal

Cabal is a frustratingly constraining tool.  Far too frequently I encountered packages that, when trying to install, would say installing this package will break a dozen others.  If not that, then I often would be notified that the dependencies could not be resolved.  If you decide that you don’t care that your other packages break and go ahead with the install, it’s a tossup as to whether or not the install will be successful, but oops, you just broke a dozen of your old packages already, so if it fails you have some cleaning up to do.  I’ve become pretty skilled at clearing out my cabal and ghc databases and invoking cabal install world to just bring my packages back to a self-consistent state after trying and failing to install a new package to explore.

For packages that leverage a cross platform library, this is frustraing but understandable.  Often this house of cards is built by taking C or C++ source meant for a Unix like build environment, installing it through MINGw32 (as the special Windows installation instructions tell you to do), then installing a package from cabal that brings in the Haskell wrappers.  It’s not pretty, but it’s kind of expected when pulling in functionality across platforms and across languages.

What I don’t understand is why cabal should ever have a problem with some packages breaking others or dependencies not being resolved.  Cabal knows what package I want to install, it knows what versions of its dependencies it needs, and those dependencies are freely available from hackage (as are their own dependencies).  You would think that this is a simple scenario with a straight forward solution – recurse and build all versions of everything as needed.  But for some reason, cabal does not allow side-by-side installation of different versions of packages.  If you have a package that requires a specific version of a dependency, you are effectively prevented from using any package that requires a different version.  The utility cabal-dev is available, which allows the set of packages to be sandboxed on a per project basis, but that is small consolation if you (like me) need to get out of hell on a single project.

Side-by-side installation of multiple versions is a known and demonstrated solution to dependency hell.  I alternate between blaming the community for not solving this problem and blaming myself for not understanding why it is so difficult.  Solutions like Haskell Platform seem to me to be treating the symptom and not the disease.

Haskell Community

The Haskell community is widely regarded as newb-friendly.  If you need newb help, the haskell IRC channel is just a click away, and in cases where I didn’t get help from the IRC channel or haskell cafe, I’ve simply selected a knowledgable member of the Haskell subreddit and messaged them directly.  No one has ever called me a newb or made me feel dumb for asking questions, even when the answer was staring me in the face in the docs and I just didn’t notice.  It’s a great community.

What has happened (albeit occasionally) is that I might get no response at all, and as my questions become more advanced and specific, the likelihood that my queries get lost in the void increases.  This is no different from any other language community.  However, the Haskell community is tiny compared to, say, Python, so it’s pretty easy to exhaust all avenues of help.  If you’re in the Haskell Windows community, it’s even smaller.

Desperation in Finding a GUI Package

In my search to find a viable GUI package, the only one that I had absolutely any success with was HGamer3d, and it’s probably not a coincidence that it is designed to work on Windows.  It was the only package that I was able to install successfully (without any headaches either), and it also had several example sources that worked effortlessly.  The problem is that is has next to nothing for documentation.  I tried modifying an example slightly only to have the executable fail on startup for unknown reasons.  Maybe I will revisit it in the near future, but I would prefer to not have to learn Ogre to decipher the HGamer3d source code.

What I wanted at the start of all of this was a GUI library or framework that supported DDS texture files.  By the end, I was willing to settle for anything that would successfully install and would display an image in any format.  So why not the web frameworks?  Yesod can’t resolve dependencies, happstack fails to load the network package (for which I found a work around online that, alas, doesn’t work), and snap fails to build the lens package due to a segfault in the generated code.  I’m sure any of these would work if I had the “right” set of packages installed, but this is not a viable solution (see above).  I can’t rip my environment apart every time I need something new.

wxHaskell is the most popular GUI toolset based on this reddit discussion, but I forgot to sacrifice a goat as step 0 of the installation process, so I had very little success with it.

Now

I feel pretty similar to the author of this reddit discussion, except I had even less success.  GLUT, GLFW, elm, reactive this, FRP that.  I feel as if I have looked at everything and found success in nothing.  Others work in Haskell GUIs all the time, so I know it is possible and that part of the blame lies on my side of the screen.

This experience has been a completely different kind of difficulty compared to learning the Haskell language.  List comprehension, immutable variables, monads, type classes…these are all well documented concepts with essentially no variability from case to case.  Read up long enough and eventually they click, and you are on your way.  The problem I face now is that I don’t know where to go from here.

Eureka! FNIStash breakthrough

My last few posts have been about my hobbyist attempts at building a muling/looting application for Torchlight 2, akin to ATMA for Diablo 2.  I call this application FNIStash.  Well…I’ve made something of a big breakthrough in understanding how (most) of the item modifiers in the save file operate.  Indeed, I now have the capability to take my shared stash file as input and generate the following as output:

GUID: 8940402106096732966
Full name:  Beast Charm
Num Enchants: 0
Item level: 90
Used Sockets: 0/0
Dmg/Armor: 4294967295/111
Num elements: 0
Mods:

+55.46508 Physical Damage

Gems:

GUID: -3941536332261843499
Full name: Commanding [ITEM] Forest Gauntlets
Num Enchants: 0
Item level: 44
Used Sockets: 0/0
Dmg/Armor: 4294967295/27
Num elements: 0
Mods:

+15.0% pet and minion Damage
+15.0% pet and minion Health

Gems:

GUID: 9067344113709864775
Full name:  Shank’s Gloves
Num Enchants: 0
Item level: 48
Used Sockets: 0/1
Dmg/Armor: 4294967295/33
Num elements: 0
Mods:

+34.199997 Ice Armor
+85.5 Fire Armor
+34.0 Dexterity Attribute bonus

Gems:

GUID: -3961984571336593215
Full name:  Pocketwatch of the Tennant
Num Enchants: 0
Item level: 31
Used Sockets: 0/1
Dmg/Armor: 4294967295/0
Num elements: 0
Mods:

+419.99997 Health
+4.5319996 Vitality Attribute bonus
+26.699999 Electric Armor
+26.699999 Fire Armor
+26.699999 Ice Armor
+26.699999 Poison Armor

Gems:

A huge hurdle in building FNIStash as thus been overcome.  There’s still a long way to go, but it’s really starting to take form.

String Keys Suck

In my previous post, I mentioned that I needed 95 MB of space memory just to run my simple test of extracting data from the Torchlight 2 PAK file.  I did some investigating to figure out what the heck was going on.  The culprit: using a FilePath (which is just a String) as the key to my map.

Prepping for profiling

To profile in Haskell with GHC, you need to compile your program with the -prof option, and throw in -auto-all to automatically add cost centers to the profile output.  You then execute the program with some additional flag to tell the runtime to collect profiling data.  After that, you can look at the resulting .prof file for nice tabular data, but I prefer graphs.  There are a few annoying steps to this whole process, so I created this batch script to handle most of it for me, which I named hprofile.  It runs the exe, generates the products, and tags the prof and graph with a description.

@echo off
%2.exe %3 %4 %5 %6 %7 %8 %9
hp2ps -e8in -c "%2.hp"
DEL %2.hp
set ID=%~1
set newname=%2(%ID%)
IF EXIST "%newname%.prof" (DEL "%newname%.prof")
RENAME "%2.prof" "%newname%.prof"
IF EXIST "%newname%.ps" (DEL "%newname%.ps")
RENAME "%2.ps" "%newname%.ps"
DEL "%2.aux"
CALL ps2pdf "%newname%.ps"
DEL "%newname%.ps"

Baseline – with String keys

Here’s the graph resulting from

>hprofile "baseline" FNIStash-Debug +RTS -p -hc
Memory usage for String keys in map

Memory usage for String keys in Map

The forText2/pakFileList is the function that generates the keys in the Map.  In this case, the keys are Strings (FilePaths).

Improvement – with Text keys

I changed the type of the key in the map from FilePath to Text.  This actually made a lot of sense since I parsed them out as Text anyway, but chose FilePath before so I could use the path handling utilities in System.FilePath.  The lookup function on the map still takes a FilePath as the key.  Now, however, the FilePath is converted to Text within the lookup function.  Here is the result.

Memory usage for Text keys in Map

Memory usage for Text keys in Map

No more runaway memory usage!  The moral of the story: avoid String.

Reading the TL2 PAK file

I’m gradually continuing to make progress with FNIStash, gradually being the key word.  It’s pretty tough to learn a new language, decode a file format, and plan a wedding at the same time.

Lately, I’ve been working on a Haskell module to decode the Torchlight 2 PAK file.  The PAK file is a single archive file that contains all the configuration and asset files for the game (well, at least most of them).  I’m hoping to read this PAK file and decode its contents so I can, for instance, grab the bitmaps for the various inventory items and use them in the GUI for FNIStash.

In writing the PAK module, I came tried a few different strategies for parsing the binary data, with varying levels of success.  I won’t describe the PAK format here because it is already well described in the readme file included in CUE’s Torchlight II packer.

Objective

The PAK.MAN file associates a file path to a particular part of the PAK file.  What I’d like to do is create a lookup table of all the files archived in the PAK file.  The key would be the file path, and the value would be the right data out of the PAK file.  I call this a PAKEntry.  So the mapping would look something like

type PAKFiles = M.Map FilePath PAKEntry

with PAK entries being parsed with the following Get operation

getPAKEntry :: Get PAKEntry
getPAKEntry = do
    hdr <- getWord32le
    decSize <- getWord32le
    encSize <- getWord32le
    encData <- getLazyByteString $ fromIntegral encSize
    return $ PAKEntry hdr decSize encSize encData

 

Attempt 1 – Using lazy Get

My first attempt used the lazy binary Get monad.  This was also lazy on my part.  My thinking here was that, since I already head a Get monad operation coded up that could parse out a PAKEntry, I could just add a line at the beginning of that operation that skips to the right part of the file.  Something like

(skip . fromIntegral) (offset-4)

with the offset being passed in as a function argument (so the type signature would be Word32 -> Get PAKEntry).  This worked, but was horrendously slow.  What this does is read all the data of the PAK file into memory up to the offset (approximately), and then reads in as much more as it needs to parse out the PAK entry.  For parsing entries near the end of the 800 MB file, this took a long time.  I used this function to construct the table:

readPAKFiles :: FilePath -> FilePath -> IO PAKFiles
readPAKFiles manFile pakFile = do
    man <- readPAKMAN manFile
    pak <- BS.readFile pakFile
    let fileList = pakFileList man
        offsetList = pakFileOffsets man
        fileOffsetList = L.zip fileList offsetList
        f offset = flip runGet pak ((getPAKEntry . fromIntegral) offset)
        mapList = L.zip fileList (L.map f offsetList) :: [(FilePath, PAKEntry)]
    return $ M.fromList mapList

Attempt 2 – Using strict Get

Attempt 2 used the strict Get monad and strict bytestrings.  The code is nearly identical to attempt 1.  Here, the problem was that the BS.readFile function read in the entire contents of the PAK file to memory, then skipped to the right location of the PAK entry (instead of streaming the file into memory as necessary as it did with lazy strings).  This was *much* faster, but required a huge 800 MB of memory to be read in just to parse out a tiny 26k PNG file (the file I was using for testing.).

Attempt 3 – Handle and lazy Get

Check out the type signature of the PAKFiles type.  It maps a FilePath to a PAKEntry.  No where in that definition is IO mentioned.  So PAKFiles cannot do any IO, by definition.  The previous two attempts have PAKFiles defined as a pure result of parsing binary data read through IO.  What I really needed is for PAKFiles to have this type signature:

type PAKFiles = M.Map FilePath (IO PAKEntry)

Now, the result of a lookup operation is an IO action that retrieves a PAKEntry.

Reading the PAK entry lazily is the behavior I wanted to keep because I didn’t know how much data I needed to read in until after I started reading the entry.  But I didn’t want to lazily read in and throw away the data in order to skip to the right part in the file.  My next attempt used handles:

readPAKFiles :: FilePath -> FilePath -> IO PAKFiles
readPAKFiles manFile pakFile = do
    man <- readPAKMAN manFile
    let fileList = pakFileList man
        offsetList = pakFileOffsets man
        fileOffsetList = L.zip fileList offsetList
        f offset = do
            h <- openBinaryFile pakFile ReadMode
            hSeek h AbsoluteSeek (fromIntegral offset - 4)
            pak <- BS.hGetContents h
            let parsedEntry = flip runGet pak getPAKEntry
            hClose h
            return parsedEntry
        mapList = L.zip fileList (L.map f offsetList) :: [(FilePath, IO PAKEntry)]
    return $ M.fromList mapList

There’s a problem with this, though.  I kept getting errors that the handle was closed before I tried to read it!

Attempt 4 – (Mostly) success with $!

The problem with attempt 3 is that runGet is executed lazily, but hOpen and hClose are not.  When runGet is finally performed, the handle is already closed.  This is easily fixed with the $! operator, which strict evaluates its argument:

readPAKFiles :: FilePath -> FilePath -> IO PAKFiles
readPAKFiles manFile pakFile = do
    man <- readPAKMAN manFile
    let fileList = pakFileList man
        offsetList = pakFileOffsets man
        f offset = do
            withBinaryFile pakFile ReadMode (\h -> do
                hSeek h AbsoluteSeek (fromIntegral offset - 4)
                pak <- BS.hGetContents h
                return $! (flip runGet pak getPAKEntry))
        mapList = L.zip fileList (L.map f offsetList) :: [(FilePath, IO PAKEntry)]
    return $ M.fromList mapList

This version of the operation is fast and uses a low amount of memory.  By “low”, I mean 95 MB to parse out a 26k file.  Something is wrong there, which is my next thing to investigate.  Maybe that will make for another post.

– Update 01/23/2013 –

This post last ended with my creating a lookup table that generated IO actions that returned PAKEntry records.  This turns out to be a horrible idea.  Every time a lookup is made, IO has to be run.  In order to do this, you need to pollute huge parts of your program with the IO monad.  Instead, I reformulated this to be pure and more forward looking – you give the function a clue of what content you want in the table (such as a file path prefix), and it will read all of the necessary files into memory right then and there.  After that, lookups can be done without the IO monad.

Building ATMA for Torchlight 2 (FNIStash)

Back in my old Diablo 2 days, there was an incredible third-party application called ATMA.  ATMA was a muling application for single player.  In Diablo 2, your characters have limited space in your inventory, so you frequently needed to either toss valuable items away or put them on a separate character (called a mule).  ATMA (short for A Tenshi Muling Application) essentially gave you a limitless inventory for your characters.  By modifying the actual Diablo 2 save files, it could “pull” items out of your character’s inventory, store them in a separate and searchable database, and then insert them back into your inventory when you wanted them.

Well, I’ve been playing a little Torchlight 2 recently and found that no similar utility currently exists.  Searching the Torchlight 2 forums reveals a few posts related to the topic, including the scrambling/descrambling algorithm used to obscure the save files, as well as the item format.  Cthon (of clockworkcore.org, I believe) is the furthest along.  He wrote a TL2 character editor that’s quite nice, so I imagine it’s only a matter of time before a nice muling app comes out as well.  The CC website says that the full file specification will be out “hopefully before the end of October.”

Well, I got tired of waiting so I started working on it myself.  Here’s an excerpt of my shared stash data dump after running my code on it:

Full name:  Labarinto Boots
Num Enchants: 1
Item level: 70
Used Sockets: 0/2
Dmg/Armor: 4294967295/51
Num elements: 0
Num mods: 4
Mods: Right [Right

Type: 32833
Name: “OFTHETURTLE ARMOR BONUS”

,Right

Type: 32833
Name: “OFTHEBEAR DAMAGE BONUS”

,Right

Type: 32833
Name: “”

,Right

Type: 33857
Name: “TRINKET_ELECTRICDEFENSE ELECTRICAL DEFENSE”

]
Footer: 0x 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The “Right” labels are a Haskell thing indicating something was achieved successfully.  In this case, it’s parsing the mods out of the file correctly.

What I eventually hope to build is a small utility similar to ATMA, which I have tentatively named FNIStash. I want to include basic searchability, and maybe at some point in the future an optimization tool that will automatically select a good set of gear based on certain criteria.  GUI maybe someday, although I’m primarily using this project as a way to learn more Haskell.

I wouldn’t count out clockworkcore.org yet.  I suspect they have something pretty well developed already but are keeping it under wraps until it’s done.

 

Installing Leksah on Windows 7

This past weekend, I convinced myself to give Leksah, the Haskell IDE written in Haskell, a try.  I’m glad I did.  Writing Haskell in it is much more enjoyable than using Notepad++ and WinGHCi side by side.  So much more enjoyable that I actually managed to sit down and write something useful!

Installing Leksah on Windows is not hard, but it is easy to mess up, as I found out.  Here is what I hope is a step-by-step guide to doing it on Windows 7.  Note that this walkthrough assumes you already have GHC installed, probably by installing the Haskell Platform.

  1. Determine your ghc version by typing “ghc –version” at a command prompt.  In my case, it was 7.0.4.
  2. Download the Leksah binary installer that matches your ghc version.  Matching of versions is not optional!
  3. Run the installer and accept the default installation location.  Mine was C:\Program Files (x86)\Leksah\
  4. Start up Leksah (you can search for it in the Start menu).  When you first start up Leksah, you will be asked to provide the path to your cabal packages.  This is so Leksah can process your packages into an explorable database.
  5. Add the packages path into the text field, then click the Add button.  You should see the path listed in the path list as confirmation of your selection.  My path was C:\Users\MyUserName\AppData\Roaming\cabal\packages.  Then click OK.
  6. Leksah will process your installed packages.  This takes a few minutes.  If it looks like Leksah performed this processing really quickly/nearly instantly, then double-check that your Leksah installer matches your ghc version.  (This was my problem for quite a while.)
  7. Leksah saves your package data in a .leksah folder in the top level of your user directory.  If you delete this folder, Leksah will revert its behavior as if it were freshly installed.  This is useful if the guide above doesn’t work for you and you need to debug some stuff but don’t want to endlessly uninstall then reinstall.

And that’s it!  If you need help on using Leksah, try this series of youtube videos.

Converting continuous filters to discrete implementation

Here’s a trick to quickly convert a continuous, strictly proper transfer function into pseudocode for a discrete time implementation.  Let’s start with a first order low pass filter:

$$\frac{Y(s)}{U(s)} = \frac{1}{\tau s + 1}$$

The first step is to cross multiply the terms (for simplicity I’ll omit the Y(s) and U(s) for the rest of this post).

$$\tau sY + Y = U \rightarrow \tau \dot{Y} + Y = U $$

Then solve the transfer function for the highest ordered derivative of $Y$ (meaning the $Y$ term with the most $\cdot$ above it), then integrate both sides of the equation until $Y$ (with no dots) becomes the left hand side.  The integration is with respect to time, but I’ve omitted that for clarity in the expression.

$$ \dot{Y} = \frac{U-Y}{\tau} \rightarrow \int{\dot{Y}} = \int{\frac{U-Y}{\tau}} \rightarrow Y = \int{\frac{U-Y}{\tau}} $$

Now replace the $=\int\left(\right)$ with a $+= dt\left(\right)$ to achieve the following pseudocode

$$ Y += dt\left( \frac{U-Y}{\tau} \right)$$

Here, $dt$ is the time between filter updates, and $Y$ is a static variable whose value persists between filter updates.  What we did is solve for the derivative of the filter output ($\dot{Y}$) and then integrated the derivative using Euler’s method to get the filter’s software implementation.

We can do the same thing with a second order filter:

$$ \frac{Y}{U} = \frac{\omega_n^2}{s^2 + 2\zeta\omega_ns + \omega_n^2} $$

$$\ddot{Y} + 2\zeta\omega_n \dot{Y} + \omega_n^2 Y = \omega_n^2 U \rightarrow \ddot{Y} = -2\zeta\omega_n \dot{Y} + \omega_n^2(U-Y) $$

Here we’ve again solved for the highest ordered derivative of $Y$.  Now what we’ll do is integrate both sides of the equation until we are again left with only $Y$ on the left hand side.

$$ \int{\ddot{Y}} = \int{-2\zeta\omega_n \dot{Y} + \omega_n^2(U-Y)} \rightarrow \dot{Y} = -2\zeta\omega_n Y + \int{\omega_n^2(U-Y)} $$

$$\int{\dot{Y}} = \int{\left(-2\zeta\omega_n Y + \int{\omega_n^2(U-Y)}\right)} \rightarrow Y = \int{\left(-2\zeta\omega_n Y + \int{\omega_n^2(U-Y)}\right)} $$

Let’s break the previous expression into two parts so only one $\int()$ appears on the right hand of any equals sign.

$$ k = \int{\omega_n^2(U-Y)} $$

$$ Y = \int{\left(-2\zeta\omega_n Y + k\right)}$$

Now we pull the same trick we did before by replacing each $=\int()$ with $+= dt()$ to arrive at the software pseudocode.

$$ k += dt\left(\omega_n^2(U-Y) \right)$$

$$ Y += dt\left(-2\zeta\omega_n Y + k \right)$$

Now we have two static variables, $k$ and $Y$, that each represent a separate state variable of the second order filter.

As an exercise for the reader (oh man, did I really just say that?), show that the second order band pass filter

$$ \frac{Y}{U} = \frac{2\zeta\omega_n s}{s^2 + 2\zeta\omega_ns + \omega_n^2} $$

has the following pseudocode implementation

$$ k += dt(-\omega_n^2Y) $$

$$ Y += dt\left(k + 2\zeta\omega_n(U – Y)\right) $$