Lz77 compression, SD0 Archive Gundam Ghiren

Extraction and unpacking of game archives and compression, encryption, obfuscation, decoding of unknown files
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Hey guys,

I have a question, I'm looking at a game archive and the structure resembles lz77, RLE.

I know this is broad..but am I on the right track, it appears that the compression is starting around x15


Image




LZ77 compression scheme.

The compressed data will be in the pattern of 1 bite that is gathered by 8 flag bit and 8 chunks.
Ex) [Flag][chunk0][chunk 1][chunk2][chunk3][chunk4][chunk5][chunk6][chunk7][flag][chunk0][chunk1]....

Each bit of the flag indicates whether the chunk was compressed or not.1 means compressed chunk, 0 will be the chunk has not been compressed and has a size of 1 bite.
#0bite represents # 0 chunk, while #7bit represents #7 bit.


There are 4 types of compressed chunk.

If the first bite of the chunk’s lower nibble has a value of 0, thats mean it is a long compression.1 means RLE, 2means Copy, the rest means compressed.
Long compressed chunks are formed with 3 bites.

If the 3bite comes in the order of [H0:L0][H1:L1][H2:L2] ,( H is upper nibble,4 bit), L is lower nibble. They are enclosed by bite. L0 is 0.

The enclosed 12 bit which is the order of H1 L1 H0, helps you to understand how much backwards you have to refer in advance starting from current cursor.

Add 0x10 to 8bit which is enclosed by H2 L2, that tells you how much bite you have to bring.

If it is 30 12 24 , after seeing L0 which is 0, you know it is a long compressed chunk . 0x123 tells you how much you have to refer backward from previous.
and (0x24 + 0x10)is how much bite you have to bring from there.

RLE chunks are formed by 2 bite.
L0 has to follow 1. Add3 to H0 tells you how many times you have to repeat.[H1:L1]tells repeating bite.
If it 21 78, that means 0x78 words was repeated 2+3 times.

Copy chunks are formed by 2+nbite.
In case L0 is 2, add0x12 to H1 L1 H0, you will know how much bite you have to copy. You just have to copy the data shows after that.

Compressed chunk is formed by 2bite.
The value of L0 same as previously brought data’s bite. H1 L1 H0 tells you how much you have to go backward from the current cursor.

There are 3 points in total.
1. The point that indicates from previous to present .
2. The point that indicates the present of compressed input file.
3. After decompressing and saving, the point that is used for output.


It reads one bite from the input point. Input point will move automatically.
After seeing the bite, it will make an assumption that it is a copy chunk.

After reading another bite, it will calculate the length of n. input point will move automatically.
for(i = 0; i < n; i++) {
c = fgetc(input point);
fputc(c, output point);
Previous [previous point ] = c; previous point++;
}

Previous size was 4kb. Since the range of previous point was 0~4095, after4095, instead of 4096, it will move back to 0.
Previous point= (previous point+ 1) % 4096;
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

Are you sure this algorithm is not implemented yet in quickbms?
Have you tried the compression scanner?
Maybe provide the original file.
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Here is an SD0 archive I've removed from the main file.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

The CLZ77 algorithm of quickbms gives a possible result but I don't think it's the correct one.
Why don't you make a C or Java or .NET implemenation of what you figured?
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Thank you for the input, I have a program from a Japanese game development website called westside, that extracts the files...
its just a headache, because everything I've tried doesn't bare fruit, both programming and bms wise

Also, for your own records, I'll give you a copy of the program.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

I just dumped the whole decompression function to use it directly in quickbms, anyway it's easy to decompile or converting it to C by hand:

Code: Select all

set MEMORY_FILE10 binary "\x83\xec\x18\x55\x57\x8b\x7c\x24\x24\x33\xc0\x33\xc9\xba\x0c\x00\x00\x00\x8a\x67\x0a\x8a\x4f\x08\x8a\x47\x09\xc1\xe0\x08\x0b\xc1\x89\x44\x24\x1c\xb8\x00\x00\x00\x00\x88\x44\x24\x0b\x88\x44\x24\x0a\x8b\xe8\x0f\x86\x72\x01\x00\x00\x53\x8b\x5c\x24\x2c\x56\x8a\x44\x24\x12\x84\xc0\x75\x0d\x8a\x04\x3a\xc6\x44\x24\x12\x08\x88\x44\x24\x13\x42\xf6\x44\x24\x13\x01\x75\x0d\x8a\x0c\x3a\x42\x88\x0c\x2b\x45\xe9\x19\x01\x00\x00\x8a\x04\x3a\x8a\x4c\x3a\x01\x88\x44\x24\x18\x83\xc2\x02\x24\x0f\x88\x4c\x24\x14\x88\x44\x24\x20\x75\x3a\x8b\x74\x24\x14\x8b\x44\x24\x18\x81\xe6\xff\x00\x00\x00\x25\xff\x00\x00\x00\xc1\xe6\x04\xc1\xe8\x04\x03\xf0\x33\xc0\x8a\x04\x3a\x83\xc0\x10\x74\x0f\x8b\xcd\x2b\xce\x45\x48\x8a\x0c\x19\x88\x4c\x2b\xff\x75\xf1\x42\xe9\xc5\x00\x00\x00\x3c\x01\x75\x48\x8b\x44\x24\x14\x8b\x4c\x24\x18\x25\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\x8d\x3c\x2b\x8a\xd8\xc1\xe9\x04\x8a\xfb\x83\xc1\x03\x8b\xc3\x89\x4c\x24\x1c\x8b\xf1\xc1\xe0\x10\x66\x8b\xc3\x8b\x5c\x24\x30\xc1\xe9\x02\xf3\xab\x8b\xce\x83\xe1\x03\xf3\xaa\x8b\x7c\x24\x2c\x8b\xc6\x03\xe8\xeb\x79\x3c\x02\x75\x3f\x8b\x44\x24\x14\x8b\x4c\x24\x18\x25\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\xc1\xe0\x04\xc1\xe9\x04\x8d\x34\x3a\x8d\x3c\x2b\x8d\x4c\x08\x12\x89\x4c\x24\x1c\x8b\xc1\xc1\xe9\x02\xf3\xa5\x8b\xc8\x03\xd0\x83\xe1\x03\x03\xe8\xf3\xa4\x8b\x7c\x24\x2c\xeb\x36\x8b\x74\x24\x14\x8b\x4c\x24\x18\x8b\x44\x24\x20\x81\xe6\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\xc1\xe6\x04\xc1\xe9\x04\x03\xf1\x25\xff\x00\x00\x00\x76\x0f\x8b\xcd\x2b\xce\x45\x48\x8a\x0c\x19\x88\x4c\x2b\xff\x75\xf1\x8a\x44\x24\x12\x8a\x4c\x24\x13\xfe\xc8\x88\x44\x24\x12\x8b\x44\x24\x24\xd0\xe9\x3b\xe8\x88\x4c\x24\x13\x0f\x82\x9e\xfe\xff\xff\x5e\x5b\x8b\xc5\x5f\x5d\x83\xc4\x18\xc3\x8b\xc5\x5f\x5d\x83\xc4\x18\xc3"
idstring "SD0\0"
get ZSIZE long
get SIZE long
savepos OFFSET  # unused because the function wants offset 0
comtype calldll "MEMORY_FILE10 0 stdcall RET #INPUT# #OUTPUT#"
get NAME basename
get EXT extension
string NAME + "_unpack."
string NAME + EXT
clog NAME 0 ZSIZE SIZE
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Dude,

you are amazing

Thank you!
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Thank you,

I was wondering if I am doing something wrong? I assumed quickbms could open the MRG files, the SD0 Files dumped.
the one for legend of the dragon I changed it to "MRG\0"

I'm new to bms, are they just totally different?...hmm wait..I answered my own question.
Last edited by lsaint on Tue Oct 18, 2016 12:57 am, edited 1 time in total.
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Ok,

I should see if I can figure this out, bms seems pretty amazing...let's see if a Newb can war game through this

Image
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

The script was just a decompressor.
The MRG format doesn't seem so simple, it's not a classical archive.
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Yes,

with a little work I figured out that the MRG's are a variant of an animation format know as MNG,
thank you for your help.

This project just gets, more and more indepth
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

I'm kind of at Empasse, good it be a graphics format "TX" represents individual Textures, and PSET sets the dimensions of
the textures.

I can't seem to find a program that can open it.
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

I have one more question

how would I loop the bms script to go through the complete file?
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

What you mean?
Isn't just one compressed file?
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

It comes from a larger archive..
meaning there are several SD0's in one MKD (the Main file), but the archive is large so I couldn't post it here

so have it pull all of the SD0's out of the main file
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

Post it on mega.nz and paste the link here.
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Hello,

here is a link to a full MRG File.

https://mega.nz/#!DRNHQJbT!eA_5HADq1ykh ... ZENH_JJ4BQ
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

http://aluigi.org/bms/gundam_gihren.bms

Now it's clear that these MKD archives are just raw containers and there are other information (including names probably) in smaller index files.
You should check the list of your files, probably there is a small file called ZZZPSP1.*
lsaint
Posts: 22
Joined: Tue Oct 04, 2016 10:55 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by lsaint »

Yup,
Last edited by lsaint on Sat Oct 22, 2016 3:44 pm, edited 1 time in total.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Lz77 compression, SD0 Archive Gundam Ghiren

Post by aluigi »

I don't see names or useful references.
Anyway it's not necessary, the scanning feature of the script does the job so no problem :)