How to deal with unknown file compression

[Cuber456]

Hello all,

I am working on reverse engineering model and texture formats in a 3DS game called Virtue's Last Reward. With some help, I've done a decent job doing so and can render them in a model viewer I coded up.

Here is Phi in my model viewer:
https://www.youtube.com/watch?v=28ubdtUBMQQ

Here is what she actually looks like in game:


Some models and textures appear in archives. They can be ripped out pretty easily. Now here is where the problem arises. Files inside of these archives are compressed. In theory I could render them in my viewer with no issue but I can't touch them because they are compressed. I don't know what to do to uncompress them.

Here are some sample files. 3 of them are compressed and 3 of them are uncompressed:
http://www.mediafire.com/download/zyac7w1fzo24e6p/Test_Files.7z

Can someone give me some pointers to help me decompress these files? I appreciate any advice I can get.

Thanks.

[aluigi]

The algorithm is LZF.
With the quickbms compressions scanner the algorithms LZF, DARKSECTOR_NOCHUNKS, FASTLZAH, EVOLUTION, UNKNOWN6 and BLOSC give the same identical correct result.

The script is the following:

Code: Select all

comtype lzf
get SIZE long
savepos OFFSET
get ZSIZE asize
math ZSIZE -= OFFSET
get NAME filename
clog NAME OFFSET ZSIZE SIZE

Note that I don't know if vittt_yellow.etc is compressed or obfuscated, but the script works perfectly with the other 2 files.

[Cuber456]

Wow, pretty amazing. You aren't kidding when you said that this script works perfectly for the XML and BM file. The BM file rendered just fine in my model viewer. I tried your script on a few other archived BM files and I can also render them.

There are a few things I need to check out for those ETC files. A while back, I went and wrote up my own program that could rip these files from archives. The program I wrote seemed to work on the archives that contained either XML or BM files. I'm wondering if perhaps my code is slipping up somewhere with the archives that contain ETC files. That could explain why the script doesn't work on them. I would find it strange otherwise if the devs, for no reason, decided to compress ETC archives with some other compression.

The other thing I should mention is this. Not even considering archives with their compression and that jazz, ETC files by themselves are actually a compression format to begin with. To be more specific, ETC stands for Ericsson Texture Compression. Virtue's Last Reward uses it's own variation(s) of ETC1 listed on this page:
http://en.wikipedia.org/wiki/Ericsson_Texture_Compression
This probably shouldn't affect the compression that the archives have on them though.

I'm wondering if I look over the quickbms.txt file, will I be able to learn how to write my own scripts? It would be very helpful to write a script that could just perform LZF on a whole folder directory instead of just one file. Archives can contain a bunch of files (like 100).

[puggsoy]

QuickBMS automatically includes the functionality to perform a script on all the files in a directory. I'm not sure if you can do it with the GUI, but if you use the command line you can do this by specifying a folder as the input instead of a file. Then it'll run the script on each file individually.

As for writing your own scripts, I would suggest looking at existing scripts, using quickbms.txt to check what each line does, and also looking at the files they work on to see how it works. It's extremely handy to know, it saves a lot of time and effort compared to writing an entire program.

[aluigi]

It's very easy to launch the script against selected files or all those in a folder and its sub-folders from the embedded GUI (double-click on quickbms.exe).
When you must select the input file you can just select multiple files with the mouse (CTRL+left to select many single files) or you can just write * or "" in the "File name" entry at the bottom of the window and it will scan the whole folder.

This mechanism is just written in the title of the window for selecting the input file

[Cuber456]

Note that I don't know if vittt_yellow.etc is compressed or obfuscated, but the script works perfectly with the other 2 files.

I'm wondering if perhaps my code is slipping up somewhere with the archives that contain ETC files. That could explain why the script doesn't work on them. I would find it strange otherwise if the devs, for no reason, decided to compress ETC archives with some other compression.

I found the issue with the ETC file. My program was actually slipping up somewhere. It's difficult to go into specifics but basically my program was ripping out the ETC files from the archive with file sizes smaller than what they actually were. No other files had this issue because they are generally smaller than ETC files. I patched up the problem in my program and when I ran it through the script, it decompressed just fine. You can try it yourself:
http://www.mediafire.com/download/moa9dxaox19g6pm/vittt_yellow.etc
So ya, I just want to apologize for that. That is something I should have double checked before giving you the files.

This mechanism is just written in the title of the window for selecting the input file

Huh... I should have figured this out sooner.

Oh and puggsoy, thanks for responding. I really am happy to be able to get any help that I can get.

Now... in theory... I should be able to render rooms from the game with models, textures and all...

[Cuber456]

So I've got some good news and one more question to ask for this thread. The good news is that I can (mostly) render other rooms and textures now. Check it out.

This image is of the same room that is in my initial post of this thread.

Here is an in-game (prerendered) image


And the same room rendered in my model viewer

Now for my last question (I think) for this thread. Some texture files don't appear to decompress properly using QuickBMS.
Here are four texture files where 01 decompresses but the rest do not

I'm pretty sure that these are actual texture files (instead of say dummy files). My first thought was that maybe the program I coded up to rip them out of their archive was failing or screwing up like before. To test this idea, I ripped these three files manually in a hex editor but I still got the same result so I think my program is ok. There is also the fact that textures decompressed after these three files should screw up as well if my program is on the fritz but that doesn't happen either.

Is there something that can be done to fix this? I'm sure that someone here is more knowledgeable on LZF than I am.

[aluigi]

Those files you uploaded can be correctly decompressed with the script I posted, no problems.

[Cuber456]

Wait what? You can decompress all of them? Using the same script that you originally posted??? Every time I try to decompress 02,03 and poster I just get an empty file. Only 01 decompresses for me. I don't understand.

• I copied the script posted in this thread to a new txt file and saved
• Ran quickbms.exe
• Opened the script
• Selected all 4 files at once
• Saved them elsewhere

Am I doing something you are not?

[aluigi]

Correction, only abMARK01.etc is decompressed.
The others can't be decompressed because they are already non-compressed, you must just remove the first 4 SIZE bytes.

[Cuber456]

Wow. You are right because I was able to convert the three problem files directly to BMP files. Still, that is really weird. These three files being uncompressed makes no sense to me. All other 63 ETC files in the archive these files came from are compressed. It doesn't make sense why the devs wouldn't compress these 3 files in the archive yet compress everything else. The only thing I can think of is that compressing the 3 would result in larger files.

Other archives did a similar thing where just a few ETC files wouldn't decompress which probably means they already are decompressed. At any rate, I think that mystery is solved.