Custom LZSS Compression [Nancy Drew ciftree]

[memyselfandfred]

A file I want to decompress and then recompress (not extract) uses a custom LZSS compression which is documented here

The game is Nancy Drew: Secrets can Kill, the file is CifTree.dat (attached) Despite the compression being documented, I'm unable to figure out how to decompress it with quickbms. I've already ran comtype scan and none were right

Any ideas on how I can approach this? I'm not sure how to convert that info into a script even looking at the lzss info in the quickbms documentation. Any help would be greatly appreciated

[Ekey]

Format: https://shimmerfairy.neocities.org/nd/ff_ciftree.html

[masagrator]

You can try to disassemble executable and decompile LZSS decompress function.
I recommend IDA as it's outputting often reliable code, even if it's hardly readable.

I have used this method for Desire game.
This code is almost 1:1, changed only sone types to be compatible with C++, not C (without extern C).
https://github.com/masagrator/NXGameScr ... cpp#L5-L68

But this method is not for recompressing.
So you're asking more a programming stuff.

[aluigi]

How can the lzss compression on https://gitlab.com/ShimmerFairy/oldhert ... b/lzss.cpp being ok?

Here the result is wrong because it's exactly the same of comtype lzss, which is totally wrong.

The only good thing there is the struct of the index for the 3 versions so I updated my script for working with them:
http://aluigi.org/bms/nancy_drew_ciftree.bms

[avifors]

Hey Aluigi. I tried the script for the 5th game in the series which is the Final Scene but I am getting this message.|

00000000 0 compressed\CIFLIST
00000000 0 compressed\
0000544e 0 compressed\
1bc70210 773390336 compressed\00000003.dat

Error: incomplete input file 0: C:\Users\ajbal\Desktop\New folder (6)\CIFTREE.DAT
Can't read 64 bytes from offset 1bc70210.
Anyway don't worry, it's possible that the BMS script has been written
to exit in this way if it's reached the end of the archive so check it
or contact its author or verify that all the files have been extracted.
Please check the following coverage information to know if it's ok.

coverage file 0 0% 296 8187692 . offset 1bc70210

Last script line before the error or that produced the error:
44 log NAME OFFSET ZSIZE

- OFFSET 0x1bc70210
- SIZE 0x2e190000
coverage file 0 0% 296 8187692 . offset 1bc70210

[SleepyBell]

I think I've got it.

Based on https://shimmerfairy.neocities.org/nd/ff_avf.html it seems that this archive format doesn't need custom decompression format, just preprocessing. The actual compression is the regular LZSS that QuickBMS handles already, but there's just an extra layer of pseudo-encryption that needs to be reversed before decompressing.

- For each byte, subtract its offset modulo 256 from itself, and modulo that result by 256 (...)
- Decompress the LZSS-compressed data.
- Convert the RGB555 color data to your format of choice.

I had success with the sample file that was attached in the initial post, one of the first image files (called "100X200" in the archive). Rendering the bytes as RGB555 colors, it was clearly a game image.

I think it might be possible to use the "Encryption math" feature of QuickBMS to implement the byte subtraction then decompress from there, but I'm new at QuickBMS and couldn't figure out how to refer to the current byte offset. It did work when I ran the QuickBMS extracted output through my own external decryption/decompression programs. (I also couldn't get "Encryption EXECUTE" to work but that's a separate issue.)

[spiritovod]

@SleepyBell: I suppose you need more plain code instead of encryption function. If you have offset for lzss block, it would be something like this:

Code: Select all

log MEMORY_FILE 0 0
goto OFFSET
for i = 0 < ZSIZE # compressed size
   savepos SUB
   get CURR byte
   xmath CURR "(CURR - (SUB % 256)) % 256"
   putvarchr MEMORY_FILE i CURR
next i
clog NAME 0 ZSIZE SIZE MEMORY_FILE # comtype lzss (should be already set), uncompressed SIZE

Though last modulo doesn't make much sense, unless math is done for signed values.

[SleepyBell]

Thanks spiritovod, something like that worked.

I added your logic to Aluigi's script and (after a bit of fiddling) got a QuickBMS script that spits out usable decompressed files from the archive file that started this thread. The relevant parts are inline below for easy reference, and the script is attached. I think I made the file switching more complicated than it needed to be but it does work.

memyselfandfred, the attached script will handle extracting the files. The reverse process will apply if you're trying to pack your changes back in: compress with lzss, then "encrypt" the compressed bytes by adding their position to their value.

Code: Select all

    for i = 0 < FILES
        #...

        # Simple pseudo-encryption as documented here:
        # https://shimmerfairy.neocities.org/nd/ff_avf.html
        savepos NEXT_HEADER_ENTRY 0
        putvarchr MEMORY_FILE ZSIZE 0 # we're going to copy the whole compressed file so we might as well pre-allocate the space
        log MEMORY_FILE 0 0

        goto OFFSET 0 SEEK_SET
        for j = 0 < ZSIZE
            get OBFUSCATED_VALUE byte 0
            xmath ACTUAL_VALUE "(OBFUSCATED_VALUE - (j % 256)) % 256"
            putvarchr MEMORY_FILE j ACTUAL_VALUE
        next j

        string DECOMP_NAME p "%s/%s" "decompressed" NAME
        open MEMORY_FILE
        clog DECOMP_NAME 0 ZSIZE SIZE MEMORY_FILE
        open 0
        goto NEXT_HEADER_ENTRY 0
    next i