Emulation of TL2 Shared Stash

A lot of cool news to report on FNIStash!

After successfully pulling png icons out of the TL2 game data, the next step was to get a GUI going.  As I’ve written about previously, GUIs on Haskell (especially windows) are a PITA.  However, after doing a clean reinstall of Haskell Platform, I was able to successfully install Ji and build the example executables.

What is Ji?

The GUI landscape on Haskell is pretty bad.  There are lots of frameworks out there, but actually installing them (much less using them) is often a big pain.  There really hasn’t been much effort lately to make these frameworks more user friendly.

Where does Haskell get a lot of attention?  The Web.  Ji is a package that takes advantage of this web attention by turning your web browser into desktop GUI.  You control the browser GUI from your Haskell application with commands to add elements, handle events, etc.  Ji achieves this by running a Javascript library on your page that transparently communicates with the Haskell server using JSON to execute commands and send event notifications.  The low latency of a desktop connection to itself makes this feasible, but technically Ji could be used for web apps that have a sufficiently fast connection (probably not a good idea, though).  It’s all automatic and requires very little set up.

For FNIStash, I needed to expand on the Ji package by enabling assignment of element IDs and lookups thereof.  I found that most of the infrastructure for this was already there but unused, which made adding the functionality pretty easy once I figured out how the package worked.  The updates went into Ji earlier this morning, but  Ji’s creator, Chris Done, has other things on his plate and wanted to unload responsibility for the package.  Heinrich Apfelmus, of reactive-banana FRP fame, volunteered to take the project over with my assistance.  That transition is still in the works.

Ok, so FNIStash…

I recently discovered how TL2 stores the location information for items.  There is a hierarchical system of containers, slots, and indices, each of which has an ID to keep track of it all.  From a raw item binary, I now know where that item is among TL2’s various item containers.  The result?

Image of working shared stash emulation

A working shared stash display!

This is a working shared stash display, albeit unpolished.  Each item is inserted into the emulated stash in its same position just like the game.  The selection tabs at the top work, so clicking on a different tab changes the class of items that are displayed.  Ji handles this all over the wire, although I admit is was a little tricky getting the click event handler written.

The error at the top indicates that there are some items in this shared stash file that I am not parsing correctly.  It’s true that there are cases my parsing logic doesn’t currently handle, mainly because I don’t understand the file format entirely.  For the time being, handling most cases is sufficient for moving development forward.

It’s actually starting to look like a real, genuine desktop app.  I’m excited!

Success in Extracting TL2 Icons

I recently got back to working on FNIStash in Haskell after some diversion into F#, and I had a great breakthrough in developing yesterday.  I successfully installed the Codec-Image-Devil package of bindings to the DevIL image library, which I then used to read the DDS files that are archived in TL2’s PAK archive.  For each one of these DDS files there also exists and IMAGESET (XML) file that defines the name and location of each icon in the DDS file.  Using this information, I wrote out each icon to disk as a PNG file.  My goal is that eventually these will be hosted through an HTML interface to simulate the TL2 stash.

I was somewhat selective about the icons I wrote to disk, and I still got out nearly 1000 icons.  Here are a few:

Eureka! FNIStash breakthrough

My last few posts have been about my hobbyist attempts at building a muling/looting application for Torchlight 2, akin to ATMA for Diablo 2.  I call this application FNIStash.  Well…I’ve made something of a big breakthrough in understanding how (most) of the item modifiers in the save file operate.  Indeed, I now have the capability to take my shared stash file as input and generate the following as output:

GUID: 8940402106096732966
Full name:  Beast Charm
Num Enchants: 0
Item level: 90
Used Sockets: 0/0
Dmg/Armor: 4294967295/111
Num elements: 0
Mods:

+55.46508 Physical Damage

Gems:

GUID: -3941536332261843499
Full name: Commanding [ITEM] Forest Gauntlets
Num Enchants: 0
Item level: 44
Used Sockets: 0/0
Dmg/Armor: 4294967295/27
Num elements: 0
Mods:

+15.0% pet and minion Damage
+15.0% pet and minion Health

Gems:

GUID: 9067344113709864775
Full name:  Shank’s Gloves
Num Enchants: 0
Item level: 48
Used Sockets: 0/1
Dmg/Armor: 4294967295/33
Num elements: 0
Mods:

+34.199997 Ice Armor
+85.5 Fire Armor
+34.0 Dexterity Attribute bonus

Gems:

GUID: -3961984571336593215
Full name:  Pocketwatch of the Tennant
Num Enchants: 0
Item level: 31
Used Sockets: 0/1
Dmg/Armor: 4294967295/0
Num elements: 0
Mods:

+419.99997 Health
+4.5319996 Vitality Attribute bonus
+26.699999 Electric Armor
+26.699999 Fire Armor
+26.699999 Ice Armor
+26.699999 Poison Armor

Gems:

A huge hurdle in building FNIStash as thus been overcome.  There’s still a long way to go, but it’s really starting to take form.

String Keys Suck

In my previous post, I mentioned that I needed 95 MB of space memory just to run my simple test of extracting data from the Torchlight 2 PAK file.  I did some investigating to figure out what the heck was going on.  The culprit: using a FilePath (which is just a String) as the key to my map.

Prepping for profiling

To profile in Haskell with GHC, you need to compile your program with the -prof option, and throw in -auto-all to automatically add cost centers to the profile output.  You then execute the program with some additional flag to tell the runtime to collect profiling data.  After that, you can look at the resulting .prof file for nice tabular data, but I prefer graphs.  There are a few annoying steps to this whole process, so I created this batch script to handle most of it for me, which I named hprofile.  It runs the exe, generates the products, and tags the prof and graph with a description.

@echo off
%2.exe %3 %4 %5 %6 %7 %8 %9
hp2ps -e8in -c "%2.hp"
DEL %2.hp
set ID=%~1
set newname=%2(%ID%)
IF EXIST "%newname%.prof" (DEL "%newname%.prof")
RENAME "%2.prof" "%newname%.prof"
IF EXIST "%newname%.ps" (DEL "%newname%.ps")
RENAME "%2.ps" "%newname%.ps"
DEL "%2.aux"
CALL ps2pdf "%newname%.ps"
DEL "%newname%.ps"

Baseline – with String keys

Here’s the graph resulting from

>hprofile "baseline" FNIStash-Debug +RTS -p -hc
Memory usage for String keys in map

Memory usage for String keys in Map

The forText2/pakFileList is the function that generates the keys in the Map.  In this case, the keys are Strings (FilePaths).

Improvement – with Text keys

I changed the type of the key in the map from FilePath to Text.  This actually made a lot of sense since I parsed them out as Text anyway, but chose FilePath before so I could use the path handling utilities in System.FilePath.  The lookup function on the map still takes a FilePath as the key.  Now, however, the FilePath is converted to Text within the lookup function.  Here is the result.

Memory usage for Text keys in Map

Memory usage for Text keys in Map

No more runaway memory usage!  The moral of the story: avoid String.

Reading the TL2 PAK file

I’m gradually continuing to make progress with FNIStash, gradually being the key word.  It’s pretty tough to learn a new language, decode a file format, and plan a wedding at the same time.

Lately, I’ve been working on a Haskell module to decode the Torchlight 2 PAK file.  The PAK file is a single archive file that contains all the configuration and asset files for the game (well, at least most of them).  I’m hoping to read this PAK file and decode its contents so I can, for instance, grab the bitmaps for the various inventory items and use them in the GUI for FNIStash.

In writing the PAK module, I came tried a few different strategies for parsing the binary data, with varying levels of success.  I won’t describe the PAK format here because it is already well described in the readme file included in CUE’s Torchlight II packer.

Objective

The PAK.MAN file associates a file path to a particular part of the PAK file.  What I’d like to do is create a lookup table of all the files archived in the PAK file.  The key would be the file path, and the value would be the right data out of the PAK file.  I call this a PAKEntry.  So the mapping would look something like

type PAKFiles = M.Map FilePath PAKEntry

with PAK entries being parsed with the following Get operation

getPAKEntry :: Get PAKEntry
getPAKEntry = do
    hdr <- getWord32le
    decSize <- getWord32le
    encSize <- getWord32le
    encData <- getLazyByteString $ fromIntegral encSize
    return $ PAKEntry hdr decSize encSize encData

 

Attempt 1 – Using lazy Get

My first attempt used the lazy binary Get monad.  This was also lazy on my part.  My thinking here was that, since I already head a Get monad operation coded up that could parse out a PAKEntry, I could just add a line at the beginning of that operation that skips to the right part of the file.  Something like

(skip . fromIntegral) (offset-4)

with the offset being passed in as a function argument (so the type signature would be Word32 -> Get PAKEntry).  This worked, but was horrendously slow.  What this does is read all the data of the PAK file into memory up to the offset (approximately), and then reads in as much more as it needs to parse out the PAK entry.  For parsing entries near the end of the 800 MB file, this took a long time.  I used this function to construct the table:

readPAKFiles :: FilePath -> FilePath -> IO PAKFiles
readPAKFiles manFile pakFile = do
    man <- readPAKMAN manFile
    pak <- BS.readFile pakFile
    let fileList = pakFileList man
        offsetList = pakFileOffsets man
        fileOffsetList = L.zip fileList offsetList
        f offset = flip runGet pak ((getPAKEntry . fromIntegral) offset)
        mapList = L.zip fileList (L.map f offsetList) :: [(FilePath, PAKEntry)]
    return $ M.fromList mapList

Attempt 2 – Using strict Get

Attempt 2 used the strict Get monad and strict bytestrings.  The code is nearly identical to attempt 1.  Here, the problem was that the BS.readFile function read in the entire contents of the PAK file to memory, then skipped to the right location of the PAK entry (instead of streaming the file into memory as necessary as it did with lazy strings).  This was *much* faster, but required a huge 800 MB of memory to be read in just to parse out a tiny 26k PNG file (the file I was using for testing.).

Attempt 3 – Handle and lazy Get

Check out the type signature of the PAKFiles type.  It maps a FilePath to a PAKEntry.  No where in that definition is IO mentioned.  So PAKFiles cannot do any IO, by definition.  The previous two attempts have PAKFiles defined as a pure result of parsing binary data read through IO.  What I really needed is for PAKFiles to have this type signature:

type PAKFiles = M.Map FilePath (IO PAKEntry)

Now, the result of a lookup operation is an IO action that retrieves a PAKEntry.

Reading the PAK entry lazily is the behavior I wanted to keep because I didn’t know how much data I needed to read in until after I started reading the entry.  But I didn’t want to lazily read in and throw away the data in order to skip to the right part in the file.  My next attempt used handles:

readPAKFiles :: FilePath -> FilePath -> IO PAKFiles
readPAKFiles manFile pakFile = do
    man <- readPAKMAN manFile
    let fileList = pakFileList man
        offsetList = pakFileOffsets man
        fileOffsetList = L.zip fileList offsetList
        f offset = do
            h <- openBinaryFile pakFile ReadMode
            hSeek h AbsoluteSeek (fromIntegral offset - 4)
            pak <- BS.hGetContents h
            let parsedEntry = flip runGet pak getPAKEntry
            hClose h
            return parsedEntry
        mapList = L.zip fileList (L.map f offsetList) :: [(FilePath, IO PAKEntry)]
    return $ M.fromList mapList

There’s a problem with this, though.  I kept getting errors that the handle was closed before I tried to read it!

Attempt 4 – (Mostly) success with $!

The problem with attempt 3 is that runGet is executed lazily, but hOpen and hClose are not.  When runGet is finally performed, the handle is already closed.  This is easily fixed with the $! operator, which strict evaluates its argument:

readPAKFiles :: FilePath -> FilePath -> IO PAKFiles
readPAKFiles manFile pakFile = do
    man <- readPAKMAN manFile
    let fileList = pakFileList man
        offsetList = pakFileOffsets man
        f offset = do
            withBinaryFile pakFile ReadMode (\h -> do
                hSeek h AbsoluteSeek (fromIntegral offset - 4)
                pak <- BS.hGetContents h
                return $! (flip runGet pak getPAKEntry))
        mapList = L.zip fileList (L.map f offsetList) :: [(FilePath, IO PAKEntry)]
    return $ M.fromList mapList

This version of the operation is fast and uses a low amount of memory.  By “low”, I mean 95 MB to parse out a 26k file.  Something is wrong there, which is my next thing to investigate.  Maybe that will make for another post.

– Update 01/23/2013 –

This post last ended with my creating a lookup table that generated IO actions that returned PAKEntry records.  This turns out to be a horrible idea.  Every time a lookup is made, IO has to be run.  In order to do this, you need to pollute huge parts of your program with the IO monad.  Instead, I reformulated this to be pure and more forward looking – you give the function a clue of what content you want in the table (such as a file path prefix), and it will read all of the necessary files into memory right then and there.  After that, lookups can be done without the IO monad.

Building ATMA for Torchlight 2 (FNIStash)

Back in my old Diablo 2 days, there was an incredible third-party application called ATMA.  ATMA was a muling application for single player.  In Diablo 2, your characters have limited space in your inventory, so you frequently needed to either toss valuable items away or put them on a separate character (called a mule).  ATMA (short for A Tenshi Muling Application) essentially gave you a limitless inventory for your characters.  By modifying the actual Diablo 2 save files, it could “pull” items out of your character’s inventory, store them in a separate and searchable database, and then insert them back into your inventory when you wanted them.

Well, I’ve been playing a little Torchlight 2 recently and found that no similar utility currently exists.  Searching the Torchlight 2 forums reveals a few posts related to the topic, including the scrambling/descrambling algorithm used to obscure the save files, as well as the item format.  Cthon (of clockworkcore.org, I believe) is the furthest along.  He wrote a TL2 character editor that’s quite nice, so I imagine it’s only a matter of time before a nice muling app comes out as well.  The CC website says that the full file specification will be out “hopefully before the end of October.”

Well, I got tired of waiting so I started working on it myself.  Here’s an excerpt of my shared stash data dump after running my code on it:

Full name:  Labarinto Boots
Num Enchants: 1
Item level: 70
Used Sockets: 0/2
Dmg/Armor: 4294967295/51
Num elements: 0
Num mods: 4
Mods: Right [Right

Type: 32833
Name: “OFTHETURTLE ARMOR BONUS”

,Right

Type: 32833
Name: “OFTHEBEAR DAMAGE BONUS”

,Right

Type: 32833
Name: “”

,Right

Type: 33857
Name: “TRINKET_ELECTRICDEFENSE ELECTRICAL DEFENSE”

]
Footer: 0x 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The “Right” labels are a Haskell thing indicating something was achieved successfully.  In this case, it’s parsing the mods out of the file correctly.

What I eventually hope to build is a small utility similar to ATMA, which I have tentatively named FNIStash. I want to include basic searchability, and maybe at some point in the future an optimization tool that will automatically select a good set of gear based on certain criteria.  GUI maybe someday, although I’m primarily using this project as a way to learn more Haskell.

I wouldn’t count out clockworkcore.org yet.  I suspect they have something pretty well developed already but are keeping it under wraps until it’s done.