Lz77 compression, SD0 Archive Gundam Ghiren
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Lz77 compression, SD0 Archive Gundam Ghiren
Hey guys,
I have a question, I'm looking at a game archive and the structure resembles lz77, RLE.
I know this is broad..but am I on the right track, it appears that the compression is starting around x15
LZ77 compression scheme.
The compressed data will be in the pattern of 1 bite that is gathered by 8 flag bit and 8 chunks.
Ex) [Flag][chunk0][chunk 1][chunk2][chunk3][chunk4][chunk5][chunk6][chunk7][flag][chunk0][chunk1]....
Each bit of the flag indicates whether the chunk was compressed or not.1 means compressed chunk, 0 will be the chunk has not been compressed and has a size of 1 bite.
#0bite represents # 0 chunk, while #7bit represents #7 bit.
There are 4 types of compressed chunk.
If the first bite of the chunk’s lower nibble has a value of 0, thats mean it is a long compression.1 means RLE, 2means Copy, the rest means compressed.
Long compressed chunks are formed with 3 bites.
If the 3bite comes in the order of [H0:L0][H1:L1][H2:L2] ,( H is upper nibble,4 bit), L is lower nibble. They are enclosed by bite. L0 is 0.
The enclosed 12 bit which is the order of H1 L1 H0, helps you to understand how much backwards you have to refer in advance starting from current cursor.
Add 0x10 to 8bit which is enclosed by H2 L2, that tells you how much bite you have to bring.
If it is 30 12 24 , after seeing L0 which is 0, you know it is a long compressed chunk . 0x123 tells you how much you have to refer backward from previous.
and (0x24 + 0x10)is how much bite you have to bring from there.
RLE chunks are formed by 2 bite.
L0 has to follow 1. Add3 to H0 tells you how many times you have to repeat.[H1:L1]tells repeating bite.
If it 21 78, that means 0x78 words was repeated 2+3 times.
Copy chunks are formed by 2+nbite.
In case L0 is 2, add0x12 to H1 L1 H0, you will know how much bite you have to copy. You just have to copy the data shows after that.
Compressed chunk is formed by 2bite.
The value of L0 same as previously brought data’s bite. H1 L1 H0 tells you how much you have to go backward from the current cursor.
There are 3 points in total.
1. The point that indicates from previous to present .
2. The point that indicates the present of compressed input file.
3. After decompressing and saving, the point that is used for output.
It reads one bite from the input point. Input point will move automatically.
After seeing the bite, it will make an assumption that it is a copy chunk.
After reading another bite, it will calculate the length of n. input point will move automatically.
for(i = 0; i < n; i++) {
c = fgetc(input point);
fputc(c, output point);
Previous [previous point ] = c; previous point++;
}
Previous size was 4kb. Since the range of previous point was 0~4095, after4095, instead of 4096, it will move back to 0.
Previous point= (previous point+ 1) % 4096;
I have a question, I'm looking at a game archive and the structure resembles lz77, RLE.
I know this is broad..but am I on the right track, it appears that the compression is starting around x15
LZ77 compression scheme.
The compressed data will be in the pattern of 1 bite that is gathered by 8 flag bit and 8 chunks.
Ex) [Flag][chunk0][chunk 1][chunk2][chunk3][chunk4][chunk5][chunk6][chunk7][flag][chunk0][chunk1]....
Each bit of the flag indicates whether the chunk was compressed or not.1 means compressed chunk, 0 will be the chunk has not been compressed and has a size of 1 bite.
#0bite represents # 0 chunk, while #7bit represents #7 bit.
There are 4 types of compressed chunk.
If the first bite of the chunk’s lower nibble has a value of 0, thats mean it is a long compression.1 means RLE, 2means Copy, the rest means compressed.
Long compressed chunks are formed with 3 bites.
If the 3bite comes in the order of [H0:L0][H1:L1][H2:L2] ,( H is upper nibble,4 bit), L is lower nibble. They are enclosed by bite. L0 is 0.
The enclosed 12 bit which is the order of H1 L1 H0, helps you to understand how much backwards you have to refer in advance starting from current cursor.
Add 0x10 to 8bit which is enclosed by H2 L2, that tells you how much bite you have to bring.
If it is 30 12 24 , after seeing L0 which is 0, you know it is a long compressed chunk . 0x123 tells you how much you have to refer backward from previous.
and (0x24 + 0x10)is how much bite you have to bring from there.
RLE chunks are formed by 2 bite.
L0 has to follow 1. Add3 to H0 tells you how many times you have to repeat.[H1:L1]tells repeating bite.
If it 21 78, that means 0x78 words was repeated 2+3 times.
Copy chunks are formed by 2+nbite.
In case L0 is 2, add0x12 to H1 L1 H0, you will know how much bite you have to copy. You just have to copy the data shows after that.
Compressed chunk is formed by 2bite.
The value of L0 same as previously brought data’s bite. H1 L1 H0 tells you how much you have to go backward from the current cursor.
There are 3 points in total.
1. The point that indicates from previous to present .
2. The point that indicates the present of compressed input file.
3. After decompressing and saving, the point that is used for output.
It reads one bite from the input point. Input point will move automatically.
After seeing the bite, it will make an assumption that it is a copy chunk.
After reading another bite, it will calculate the length of n. input point will move automatically.
for(i = 0; i < n; i++) {
c = fgetc(input point);
fputc(c, output point);
Previous [previous point ] = c; previous point++;
}
Previous size was 4kb. Since the range of previous point was 0~4095, after4095, instead of 4096, it will move back to 0.
Previous point= (previous point+ 1) % 4096;
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Are you sure this algorithm is not implemented yet in quickbms?
Have you tried the compression scanner?
Maybe provide the original file.
Have you tried the compression scanner?
Maybe provide the original file.
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Here is an SD0 archive I've removed from the main file.
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
The CLZ77 algorithm of quickbms gives a possible result but I don't think it's the correct one.
Why don't you make a C or Java or .NET implemenation of what you figured?
Why don't you make a C or Java or .NET implemenation of what you figured?
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Thank you for the input, I have a program from a Japanese game development website called westside, that extracts the files...
its just a headache, because everything I've tried doesn't bare fruit, both programming and bms wise
Also, for your own records, I'll give you a copy of the program.
its just a headache, because everything I've tried doesn't bare fruit, both programming and bms wise
Also, for your own records, I'll give you a copy of the program.
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
I just dumped the whole decompression function to use it directly in quickbms, anyway it's easy to decompile or converting it to C by hand:
Code: Select all
set MEMORY_FILE10 binary "\x83\xec\x18\x55\x57\x8b\x7c\x24\x24\x33\xc0\x33\xc9\xba\x0c\x00\x00\x00\x8a\x67\x0a\x8a\x4f\x08\x8a\x47\x09\xc1\xe0\x08\x0b\xc1\x89\x44\x24\x1c\xb8\x00\x00\x00\x00\x88\x44\x24\x0b\x88\x44\x24\x0a\x8b\xe8\x0f\x86\x72\x01\x00\x00\x53\x8b\x5c\x24\x2c\x56\x8a\x44\x24\x12\x84\xc0\x75\x0d\x8a\x04\x3a\xc6\x44\x24\x12\x08\x88\x44\x24\x13\x42\xf6\x44\x24\x13\x01\x75\x0d\x8a\x0c\x3a\x42\x88\x0c\x2b\x45\xe9\x19\x01\x00\x00\x8a\x04\x3a\x8a\x4c\x3a\x01\x88\x44\x24\x18\x83\xc2\x02\x24\x0f\x88\x4c\x24\x14\x88\x44\x24\x20\x75\x3a\x8b\x74\x24\x14\x8b\x44\x24\x18\x81\xe6\xff\x00\x00\x00\x25\xff\x00\x00\x00\xc1\xe6\x04\xc1\xe8\x04\x03\xf0\x33\xc0\x8a\x04\x3a\x83\xc0\x10\x74\x0f\x8b\xcd\x2b\xce\x45\x48\x8a\x0c\x19\x88\x4c\x2b\xff\x75\xf1\x42\xe9\xc5\x00\x00\x00\x3c\x01\x75\x48\x8b\x44\x24\x14\x8b\x4c\x24\x18\x25\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\x8d\x3c\x2b\x8a\xd8\xc1\xe9\x04\x8a\xfb\x83\xc1\x03\x8b\xc3\x89\x4c\x24\x1c\x8b\xf1\xc1\xe0\x10\x66\x8b\xc3\x8b\x5c\x24\x30\xc1\xe9\x02\xf3\xab\x8b\xce\x83\xe1\x03\xf3\xaa\x8b\x7c\x24\x2c\x8b\xc6\x03\xe8\xeb\x79\x3c\x02\x75\x3f\x8b\x44\x24\x14\x8b\x4c\x24\x18\x25\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\xc1\xe0\x04\xc1\xe9\x04\x8d\x34\x3a\x8d\x3c\x2b\x8d\x4c\x08\x12\x89\x4c\x24\x1c\x8b\xc1\xc1\xe9\x02\xf3\xa5\x8b\xc8\x03\xd0\x83\xe1\x03\x03\xe8\xf3\xa4\x8b\x7c\x24\x2c\xeb\x36\x8b\x74\x24\x14\x8b\x4c\x24\x18\x8b\x44\x24\x20\x81\xe6\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\xc1\xe6\x04\xc1\xe9\x04\x03\xf1\x25\xff\x00\x00\x00\x76\x0f\x8b\xcd\x2b\xce\x45\x48\x8a\x0c\x19\x88\x4c\x2b\xff\x75\xf1\x8a\x44\x24\x12\x8a\x4c\x24\x13\xfe\xc8\x88\x44\x24\x12\x8b\x44\x24\x24\xd0\xe9\x3b\xe8\x88\x4c\x24\x13\x0f\x82\x9e\xfe\xff\xff\x5e\x5b\x8b\xc5\x5f\x5d\x83\xc4\x18\xc3\x8b\xc5\x5f\x5d\x83\xc4\x18\xc3"
idstring "SD0\0"
get ZSIZE long
get SIZE long
savepos OFFSET # unused because the function wants offset 0
comtype calldll "MEMORY_FILE10 0 stdcall RET #INPUT# #OUTPUT#"
get NAME basename
get EXT extension
string NAME + "_unpack."
string NAME + EXT
clog NAME 0 ZSIZE SIZE
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Dude,
you are amazing
Thank you!
you are amazing
Thank you!
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Thank you,
I was wondering if I am doing something wrong? I assumed quickbms could open the MRG files, the SD0 Files dumped.
the one for legend of the dragon I changed it to "MRG\0"
I'm new to bms, are they just totally different?...hmm wait..I answered my own question.
I was wondering if I am doing something wrong? I assumed quickbms could open the MRG files, the SD0 Files dumped.
the one for legend of the dragon I changed it to "MRG\0"
I'm new to bms, are they just totally different?...hmm wait..I answered my own question.
Last edited by lsaint on Tue Oct 18, 2016 12:57 am, edited 1 time in total.
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Ok,
I should see if I can figure this out, bms seems pretty amazing...let's see if a Newb can war game through this
I should see if I can figure this out, bms seems pretty amazing...let's see if a Newb can war game through this
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
The script was just a decompressor.
The MRG format doesn't seem so simple, it's not a classical archive.
The MRG format doesn't seem so simple, it's not a classical archive.
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Yes,
with a little work I figured out that the MRG's are a variant of an animation format know as MNG,
thank you for your help.
This project just gets, more and more indepth
with a little work I figured out that the MRG's are a variant of an animation format know as MNG,
thank you for your help.
This project just gets, more and more indepth
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
I'm kind of at Empasse, good it be a graphics format "TX" represents individual Textures, and PSET sets the dimensions of
the textures.
I can't seem to find a program that can open it.
the textures.
I can't seem to find a program that can open it.
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
I have one more question
how would I loop the bms script to go through the complete file?
how would I loop the bms script to go through the complete file?
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
What you mean?
Isn't just one compressed file?
Isn't just one compressed file?
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
It comes from a larger archive..
meaning there are several SD0's in one MKD (the Main file), but the archive is large so I couldn't post it here
so have it pull all of the SD0's out of the main file
meaning there are several SD0's in one MKD (the Main file), but the archive is large so I couldn't post it here
so have it pull all of the SD0's out of the main file
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Post it on mega.nz and paste the link here.
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
http://aluigi.org/bms/gundam_gihren.bms
Now it's clear that these MKD archives are just raw containers and there are other information (including names probably) in smaller index files.
You should check the list of your files, probably there is a small file called ZZZPSP1.*
Now it's clear that these MKD archives are just raw containers and there are other information (including names probably) in smaller index files.
You should check the list of your files, probably there is a small file called ZZZPSP1.*
-
- Posts: 22
- Joined: Tue Oct 04, 2016 10:55 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
Yup,
Last edited by lsaint on Sat Oct 22, 2016 3:44 pm, edited 1 time in total.
-
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: Lz77 compression, SD0 Archive Gundam Ghiren
I don't see names or useful references.
Anyway it's not necessary, the scanning feature of the script does the job so no problem
Anyway it's not necessary, the scanning feature of the script does the job so no problem