Offset file unzipper (offzip)

Do you know a tool, link or website for working on a specific game files or to help game research? Let's collect them here!
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Offset file unzipper (offzip)

Post by aluigi »

http://aluigi.org/mytoolz.htm#offzip

Offzip is probably very known now, anyway it's a decompressor and scanner of zlib and deflate data... the classical algorithm used in the ZIP files and in the majority of games.

I have just improved the tool in the current 0.3.6 version, the following is the changelog:
added the -c option that allows to guess and dump the chunked files, option -D to specify dictionary, -d to visualize the hexdump of the data before and after the compressed streams, statistics information, offset where the compressed streams ends, amount of bytes between the current compressed stream and the previous one, zlib header and crc information, updated extensions guesser (strnicmp fix for Linux)

From the point of view of who makes the scripts there are the ending offset and the amount of space between the streams that are very useful.

One of the most interesting features is just the -c option to guess the chunked files, so those files that are splitted in many compressed and non-compressed pieces.
Imagine to have a file splitted in the following way:

Code: Select all

Offset      Compressed    Uncompressed
0x00000000: 0x00001000 -> 0x00004000
0x00001000: 0x00001000 -> 0x00004000
0x00002000: #no compressed data here
0x00003000: 0x00001000 -> 0x00004000
0x00004000: 0x00000080 -> 0x00000100
In this situation will be enough to use "offzip -a -c 0x4000" for being able to dump the original file.
Basically the tool thinks that the last chunk is the one smaller than the uncompressed chunk size (0x4000 in the example) and it will dump the non-compressed data if the space between the previous and next compressed stream is equal to the uncompressed chunk size.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Offset file unzipper (offzip)

Post by aluigi »

I have just released Offzip 0.4.
The main bigger feature is the -r option to reimport the extracted files.
It works just like in quickbms so no need of presentations :D

This is the changelog:
-r option for reimporting the extracted files like in QuickBMS (it uses the Zopfli library), the previous -r option has been renamed -R, some changes to the runtime help, it's no longer needed to specify the output folder and offset, option to automatically overwrite the output files, -1 now uses the output filename if specified, added a Makefile for Linux, renamed from Offset file unzipper to Offzip
Savage
Posts: 176
Joined: Thu Oct 02, 2014 4:58 pm

Re: Offset file unzipper (offzip)

Post by Savage »

Testing the tool, i had some issues. :D
spider91
Posts: 233
Joined: Sun Aug 24, 2014 5:26 pm

Re: Offset file unzipper (offzip)

Post by spider91 »

Can you add dump of compressed data? I need to dump all zlib blocks without decompression.

upd.

I'm blind, it's already realized with -A option :mrgreen:
spider91
Posts: 233
Joined: Sun Aug 24, 2014 5:26 pm

Re: Offset file unzipper (offzip)

Post by spider91 »

Names for compressed data dumped from big files are invalid. They are only 8 characters offsets, but should be 16 characters.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Offset file unzipper (offzip)

Post by aluigi »

Why? Offset 0x11223344 is 8 chars.
Offsets bigger than 4gb should be generated correctly if I remember correctly.
spider91
Posts: 233
Joined: Sun Aug 24, 2014 5:26 pm

Re: Offset file unzipper (offzip)

Post by spider91 »

Nope, they are not correct. I have last zlib data on offset 0x1a9a5e004, but file was named a9a5e004.dat

And it doesn't work propperly for scanning too

Image
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Offset file unzipper (offzip)

Post by aluigi »

I see.
Basically the tool is ready for 64bit variables and file operations but some points are hardcoded to 32bit (name creation) or have a bug like readbase that uses "int" instead of i64.
Added in my TODO list.
Why are you using offzip on a so huge file? What game is that?
spider91
Posts: 233
Joined: Sun Aug 24, 2014 5:26 pm

Re: Offset file unzipper (offzip)

Post by spider91 »

It's Darksiders II Deathinitive Edition. Will wait for update, thanks.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Offset file unzipper (offzip)

Post by aluigi »

don't exist extractors/scripts for it?
spider91
Posts: 233
Joined: Sun Aug 24, 2014 5:26 pm

Re: Offset file unzipper (offzip)

Post by spider91 »

Nope. And i don't want to extract it, i need compressed blocks and their offsets to reimport them back.
spider91
Posts: 233
Joined: Sun Aug 24, 2014 5:26 pm

Re: Offset file unzipper (offzip)

Post by spider91 »

I've found another bug. Darksiders II Deathinitive Edition, file maps.upak (2.87gb). Offzip dumped compressed (-A option) file named 00001004.bod with size 4271842 bytes. I've copied block from same offset and with same size from original file and files are different.

On left side file that offzip dumped, on right original block.
http://puu.sh/lnEzY/86cd042225.png

Here is a sample. Even if you dump this file now with -A it have same size but different CRC.

http://www29.zippyshare.com/v/5tfzYga0/file.html
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Offset file unzipper (offzip)

Post by aluigi »

Ok, I will check it.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Offset file unzipper (offzip)

Post by aluigi »

Sorry for necroposting but I don't think I can replicate the issue, just made a simple test (garbage+zlib+garbage) since I no longer have the sample you provided.