STAR OCEAN -anamnesis- (tri-ace New SLZ?)

[valkyrie-ace]

STAR OCEAN anamnesis is Mobile Game from Tri-ace.
I try file extract, but I can't
Files extensions looks like "Heaven x Inferno", however, header's not "SLZ", there's "ADLD"

I attached some sample files.
have any idea and can you help me?

[aluigi]

It uses a xor obfuscation with a 32bit number (converted to a 8 bytes hex string), I don't know how the key is generated but we can easily guess it
I have updated the script to automatically doing the job but remember that SLZ type 7 used in 2 of your samples is not supported yet.
Script 0.2.4:
http://aluigi.org/bms/slz.bms

[valkyrie-ace]

great work aluigi.
I have question, FIA and FIS header files, can open any softwares?
They are still unique algorithm compression files?
and will you support type 7?

[aluigi]

I don't know about FIA/FIS.
I don't have the code for the type 7 compression, once someone will provide it I will implement it in quickbms.

[akderebur]

The compression method for type 7 seems to be zstd. I have disassembled the game library to check the type 7 code, and there are calls to methods like "DecompressZSTD". Here is the pseudocode : https://pastebin.com/J39hFdr2

. I hope you can take a look at this sometime. In the bms script I have tried setting the comtype to ZSTD/ZSTD_COMPRESS, but it didn't work. New algorithm?

Edit : The zstd version that is used is 1.3.4.

[aluigi]

zstd streams have a recognizable 4 bytes magic so it's easy to guess.
"comtype zstd" automatically supports all the existent zstd versions in case you want to try.
If you don't see the zstd magic it probably means it's encrypted

[akderebur]

zstd streams have a recognizable 4 bytes magic so it's easy to guess.

Yea, you were right. There are many occurrences of the magic, so the file is divided into multiple chunks with zstd streams. Before each magic there is also a 2 byte offset to the next chunk.

I don't know how to handle this in bms. I ended up finding a zstd library and writing a program to read and decompress the chunks. The output is correct, so I can confirm that type 7 is indeed zstd. Thanks for the help.

[aluigi]

It looks like a customized zstd, in fact the magic is not 0xFD2FB528 but 0xFD0FB528.
Even fixing the byte and using comtype zstd with type 7 will not work.
So first you have to solve this mistery

[akderebur]

So first you have to solve this mistery

Already did

The problem is with the slz script actually. It doesn't fully decrypt the "ADLD" files. You have to xor the every 5th byte of 8 byte iterations with 0x20 to have the correct output. I noticed this while working on the textures, you can see the full discussion here : http://forum.xentax.com/viewtopic.php?p=142494#p142494

After doing the xor stuff, the file is decrypted properly and the zstd magics will be correct too. It is standard zstd.

[aluigi]

That's exactly what I tried before posting using one of the provided files for forcing the magic to match but it still gave errors and the obtained key had an invalid char at byte 5 (a '4' become 0x14)

[akderebur]

Weird. Here is how I do it. I have attached 2 bms scripts. First I run the "slz.bms" on the "ADLD" file to get the slz output. Then I use the "slzxor.bms" on the slz file. Maybe you can pack them into a single script.

Anyway at that point you should have the correct output. Still like I said I couldn't get it to decompress with quickbms, but I thought it is because of my lack of knowledge. I managed to get it decompress with a zstd library, so I can guarantee that the slz scripts produce the correct output.

[aluigi]

Eh no, as already said it still doesn't work:

Code: Select all

Info:  algorithm   478
       offset      00000022
       input size  0x0000302e 12334
       output size 0x00004440 17472
       result      0xffffffb8 -72

zstd works perfectly as tested with many other zstd compressed streams so if quickbms returns an error I'm sure there is something wrong with the data. Additionally the key has no sense with that invalid byte at position 5.
Keep me update.

[akderebur]

I think it will be easier if we both look at the same file. I am working with models atm, so I have just tested this with asf model files. Here is the model file of one of the characters (Myuria) that uses type 7 : http://www.mediafire.com/file/4jhficeh9 ... p.rar/file

I have included various states of the file numbered (1) to (4) :
1 - Original file with ADLD header
2 - Output of "slz.bms", not xored yet
3 - Output of "slzxor.bms", just xored
4 - Output after zstd decompression

I thought maybe this way you might be able to see in which step there is a problem.

Can you also post the file that you tried to decompress, and the script?

[aluigi]

ZSTD_decompress_usingDict (without dictionary obviously, so it gets ignored) returns -72 so there is something invalid in that stream.
quickbms uses zstd 1.3.4
I can get the data only if I ignore the error with the -e option of quickbms.
Script used:

Code: Select all

comtype zstd ""
goto 0x20
get ZSIZE short
savepos OFFSET
clog "test.dat" OFFSET ZSIZE 0x10000

What zstd function are you using for decompression and what's its return value on the first chunk?

[akderebur]

I am using this library : https://github.com/bp74/Zstandard.Net, which is basically a wrapper for the native zstd library. Instead of invoking direct decompression methods, it uses stream related methods like "ZSTD_createDStream" and "ZSTD_decompressStream". I don't know much about zstd, but do you think that might be the main difference?

[aluigi]

I tried to write a simple tool from scratch that uses ZSTD_decompress on the 0x2b28 bytes of the sample and the result is the same: -72.
I got the same result even with the other one-chunk samples so it's excluded anything related to streams, chunks, quickbms, api and so on.
-72 means ZSTD_error_srcSize_wrong

So what I did was very simple, tried to subtract 1 from the chunk size in a loop till I got a valid return value and it worked:
chunk_size - 1

Probably that .NET wrappers ignores errors or you didn't check them or the state of the decompression in your program, that's why you didn't see it.

Now the mistery is finally solved... except for the meaning of that 0x00 byte at the end of each chunk that means nothing.

Personal opinion: why this *** zstd library returns an error if the input size is bigger than the real compressed size? mah senseless...

[akderebur]

Probably that .NET wrappers ignores errors

No, it actually checks for error codes and throws an exception if there is one.

except for the meaning of that 0x00 byte at the end of each chunk that means nothing.

Not every chunk ends with 0x00 though. Check out the chunk starting at 0x4AD2 (last byte at 9CFD), it ends with 0x01.

For the chunks that end with 0x00 I can also use chunkSize - 1, but otherwise the data won't get decompressed properly or not at all. I am not sure if chunkSize - 1 is the right way to go, but if it works out that way in quickbms then it is fine I guess.

[aluigi]

In fact for those chunks that have the last byte different than 0 zstd wants the whole chunk size.
Long story short: zstd is acting in a very confusing way (*edited*).
The .NET wrapper uses ZSTD_decompressStream instead of ZSTD_decompress that probably is not affected by the "bug" of the size.

Unfortunately there is no way to bypass that error returned by ZSTD_decompress because ZSTD_error_srcSize_wrong is returned in any case the compressed size is not the exact one it expects, so both if the provided data is shorter (correct behaviour) and longer (wrong behaviour that leads to this bug).

I will evaluate if using ZSTD_decompressStream may be a good solution for the future but I don't like to waste time in this way.
More probably I will just ignore the -72 error and return the full decompressed size which is wrong too.

I'm curious to know how the game developers obtained that compressed data with the additional 0x00 byte at the end.

[akderebur]

Yea, zstd seems a bit weird. Still it seems to perform really good, or at least that is what it says on its site

I feel like it might get more popular. For example with SOA most of the older character asf files (like a year ago) were using slz compression types from the earlier games, but now almost all character asfs are compressed with type 7/zstd. It might indeed be a good idea to implement ZSTD_decompressStream for the long run.

[aluigi]

yeah zstd is gaining popularity in games.

The problem is that ZSTD_decompressStream and ZSTD_decompress are not exactly the "same".
ZSTD_decompress is a wrapper for the multiframe function and internally their 2 low-level functions ZSTD_decompressStream->ZSTD_decompressContinue and ZSTD_decompress->ZSTD_decompressFrame have no other low-level function in common.
Basically the code is redundant as clearly visible from the 2 occurrencies of "bt_compressed"... very bad idea.