QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[aluigi]

@haiwood
Please no off-topic.
I check most of the topics on the forum and if something is in the wrong section or lacks some details I fix it automatically.
And if I don't reply to a topic it means I don't have a positive solution at the moment.

[WhoKnows]

I decided to go back and extract/decrypt my Sega Genesis ROMS from my gold edition CD and Steam for archival purposes.

When I ran the 2 scripts(Sega_classics.bms and Sega_classics_gold.bms) on the QuickBMS page in 0.9.0 every file decrypted was incorrect. I ran the same exact scripts with old versions 0.7.5 and 0.7.3 and both correctly decrypted the .Pak files so I know the problem lies somewhere in the newer version. But I don't know what version the bug started in.

Additionally, I decided to compare the extracted files with a hex editor. What I found was that 0.9.0 added 3 bytes to the beginning of every ROM file decrypted. Every ROM started with these 3 bytes "EF BB BF" added. If you deleted these 3 bytes and saved, the edited files were identical to the versions created by the older QuickBMS versions.

[aluigi]

Thanks for the feedback. Can you upload one of these roms so I can check it?

Some weeks ago I noticed the following post on reddit that may be related to the same bug (I have to check it but don't know if I will have all the files for the test):
https://www.reddit.com/r/FFXVPCmods/com ... _not_work/

[WhoKnows]

Thanks for the feedback. Can you upload one of these roms so I can check it?

There is still an encrypted ROM from my gold edition in this previous thread: viewtopic.php?f=9&t=2101

It uses the script you added in that thread(sega_classics_gold.bms). The produced file was titled correctly regardless of which version of QuickBMS I used. If an older version of QuickBMS is used(I used .75) it should produce a use-able ROM. If the script is run in .90 it should create a ROM that is 3 bytes more(and they should be the bytes I posted above, at least that's what every ROM had added to it for me.) If I deleted the first 3 bytes of the .90 ROM with a hex editor it ended up the same as the ROM created with the older version.

From my testing, a ROM extracted from a Steam version .pak with the "sega_classics.bms" script was the same as a ROM extracted from a Gold edition .pak with the gold script(provided both paks were the same underlying game). This was the case as long as the same version of QuickBMS was used(I.E. the game extracted from .90 would be the same whether Steam or Gold edition(including the 3 extra bytes), and if .75 was used the steam and Gold would be the same as well).

[aluigi]

Uhmmm I tried sega_classics_gold.bms with the g0009.pak sample you provided in that topic and the result is the same in quickbms 0.9.0 or 0.7.7 or 0.7.1.
I need an exact way to replicate the problem.

[aluigi]

Ah, then do also another test.
Open quickbms.exe (0.9.0) with a hex editor, go at offset 0x95c686 and replace the byte 0x2c with 0x00.
Retry the script and let me know if now the result is good or wrong.

[WhoKnows]

Ah, then do also another test.
Open quickbms.exe (0.9.0) with a hex editor, go at offset 0x95c686 and replace the byte 0x2c with 0x00.
Retry the script and let me know if now the result is good or wrong.

The Hex change seems to have worked.

As far as how I was running it, I was using WINE command line "quickbms sega_classics_gold.bms g0009.pak"
The hex change you said though seems to have fixed it.

[aluigi]

Perfect, now I know what was the problem

[StreamThread]

NameCRC command always return a empty string, if using Temporary or Memory file as listfile.

[aluigi]

Could you please provide an example?

Here it seems to work well:

Code: Select all

set MEMORY_FILE10 binary "string1\nstring2\nmy_message\nstring3\n"
math NAME_CRC = 0x3c449c53

#test1
namecrc DUMMY 0 MEMORY_FILE10 32 0x04c11db7 "32 -1 -1 0 1 0"
namecrc NAME NAME_CRC

#test2
namecrc NAME NAME_CRC MEMORY_FILE10 32 0x04c11db7 "32 -1 -1 0 1 0"

print "%NAME%"

[Ulibos]

Hi aluigi! First of all, your projects rock! Your work on integrating so much compression algos in quickbms is beyond appreciation!
I've been fiddling around with data from R6 Siege (they changed their packaging again, old scripts don't work anymore). I know that compressed chunks with compression type 0x1 are in zstd. I successfully uncompressed them with python binding of this lib. However, quickbms fails to decode those pieces. In my case it fails with calculated uncompressed length -20. I suppose, your implementation may have fell a bit behind from official releases.
Also, I managed to find lots of Export signatures from ZSTD in the exe of the game with x64dbg, but signsrch doesn't find any traces of ZSTD. Maybe new signatures?
If needed, I can provide you a sample chunk that quickbms fails to decompress. Or any other data you need.

[aluigi]

zstd requires that both the compressed and uncompressed size are correct otherwise it returns the error -72 (ZSTD_error_srcSize_wrong), there is a long and detailed discussion here: viewtopic.php?p=37343#p37343

If you get the error/length -20 then it's the error ZSTD_error_corruption_detected that can be invoked in tons of places inside the zstd library, so it's impossible to know what caused it except that it's some malformed byte in the compressed stream.
quickbms simply calls the standard function ZSTD_decompress on the provided data so there is not much room for errors on its side

It's highly possible that the python wrapper adopts ZSTD_decompressStream instead of ZSTD_decompress, just like the .NET one described in the linked discussion.

The compressed chunk, the expected decompressed size and the decompressed chunk may be useful, yes.

[Ulibos]

Dammit... I started gathering the data for you and noticed that comp_size from .forge and actual file's size differ. It turned out I was writing out files in text mode, not binary. So those extra bytes broke the data...
Now, after reexporting that chunk quickbms unpacked it with no problems at all. So false alarm here. Guess I should sleep more before coding and check my data before posting. xD
The only thing that remains is lack of zstd table's signatures in signsrch, but that's kinda out of topic for this thread.
Anyways, thank you for your time and effort! Complementi a maestro!

[aluigi]

Eh signsrch would need some new signatures in the database but the problem is collecting all those that have been introduced (or where missing) since the last database update made some years ago.

In the meantime if you need support for zstd feel free to add the following to signsrch.sig:

Code: Select all

################################################################################
TITLE
    zstd v0.1
TYPE
    32
DATA
    0x1eb52ffd

################################################################################
TITLE
    zstd v0.2
TYPE
    32
DATA
    0x22b52ffd

################################################################################
TITLE
    zstd v0.3
TYPE
    32
DATA
    0x23b52ffd

################################################################################
TITLE
    zstd v0.4
TYPE
    32
DATA
    0x24b52ffd

################################################################################
TITLE
    zstd v0.5
TYPE
    32
DATA
    0x25b52ffd

################################################################################
TITLE
    zstd v0.6
TYPE
    32
DATA
    0x26b52ffd

################################################################################
TITLE
    zstd v0.7
TYPE
    32
DATA
    0x27b52ffd

################################################################################
TITLE
    zstd v0.8
TYPE
    32
DATA
    0x28b52ffd

[MITA996]

Alugi, I've tried to extract latest fifa online 3. And i get this message:
the uncompressed data (-1) is bigger than the allocated buffer (7455533) Last script line before the error or that produced the error:
55 clog "" OFFSET ZSIZE SIZE
Please help

[Ulibos]

Thanks for the signaturess! Idk, maybe if you're interested, I can provide you a dict table for Titanfall2 compression algo. I found it's offset. From what I know it is lzma based, but not the exact copy. Alltho I didn't bother to check exactly due to lack of experience at the time. I also have an offset of an actual decompress function. Unfortunately, I haven't finished it's disassembly. I made a python port that works but only partially. There are still some errors in a branch or two that cause it to fail on certain archives.

[aluigi]

@MITA996
This topic is not for games.
The topics for that game are viewtopic.php?t=1000 and viewtopic.php?t=3036
Just remember that raw_ea_10fb_dumper.bms is prone to false positives and there are no solutions for that.

@Ulibos
Have you already tried with the comtype_scan.bat/bms scanner?
If you are sure that it's a customized lzma (or lzma2) then it wouldn't be a simple job, currently I prefer to stay away from additional work

[Ulibos]

I did check Titanfall2 yesterday. I ran signsrch and didn't find any meaningful clues about compression algos in that binary. It is quite possible tho that inited vars are packed because I later added the signatures I have from the game and it didn't find them either. Or I did smth wrong...
I also revisited my data. The archives are packed at bit level. Default value mask is 0xffffff, i.e. at least integers are encoded in 24 bits instead of 32. Further more, archive header is also encoded in such way. It goes like this:

$arg4 is a function's arg, by default it equals 0xffffff, i.e. a bit trimming mask

$b1: 6 bits - encodes length of next block in bits
$b2: $b1 bits - packed size (it's encoded in a fancy way. Irc basically you read $b1 bits long block and slap extra bit on top. Example: $b1=4, $b2=0b0010, result=0b10010)
$b3: bitlen($arg4) bits - length of trimming mask in bits (i.e. how long (in bits) ints are in this package)
$b4: bitlen($arg4) bits - another mask, generally $b4==$b3
some skipped space. there is a padding of 0xD bits, but the resulting shift is then rounded to length in bytes. Irc in some edge cases you can get extra byte padded depending on previous blocks length. Will need to lookup the math later.
$b5: bitlen($b3 or $b4) - unpacked size
As a result of all this stuff actual data starts from like byte 14, bit 5, so you have all your data bit shifted and need to account for that. TF2 reads blocks and than stitches them with bit masks. I don't know for sure, but it seems like lzma doesn't do such hardcore bit *SPAM*, so I quess it's a proprietary compression algo.
Please keep in mind, I'm writing this data from my memory so some blocks's sizes or bit masks might be slightly wrong. Actual decoded data is at home.

[StreamThread]

Strange thing. In the last two versions: 8.4.0 - 9.0.0, ScanDir command return a memory allocation error (line 678) when any filter is specified. Even for one .txt in folder.

[aluigi]

I guess that feature has been never tested since there were at least 2 bugs in it
Will be fixed in 0.9.1