QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[aluigi]

8 maximum operators.

The differences between the two executables are mainly:
- 4gb_files is suggested to be used on files bigger than 4gb
- 4gb_files is suggested to be used on files containing files bigger than 2Gb
- 4gb_files can read 64bit fields while quickbms.exe only 32bit fields (64bits are truncated, 0x1122334455667788 is read as 0x55667788)
- 4gb_files may not work correctly in some rare situations

My suggestion is to use ever quickbms.exe, while 4gb_files must be used only when requested by the script or by the author of the script

[Shokoniraya]

i always using 4g exe, i mean, is there any problem with that? (My suggestion is to use ever quickbms.exe: but why? i always usin that)
i just want to ask: im getting problem if i just using 4g exe always?

[chrrox]

I can't get the comtype ALZSS to work.
sample file

[aluigi]

If you have ever used quickbms_4gb_files it means it works very well

The problems happen only in some rare cases like the following script:

Code: Select all

get VAR long # for example 0x88888888
if VAR < 0
    ...

For quickbms.exe VAR is < 0, while for quickbms_4gb_files.exe it's bigger.
That's why exist also signed_long which is valid with both.

I guess it may also have some problems with some dll with calldll but it's really very rare (I'm not even sure).

[aluigi]

@chrrox
Honestly I guess I didn't test the code, not even sure if I had samples.
The code is correctly called and this is the script but the output is 0 which means that it doesn't work as expected:

Code: Select all

comtype alzss
idstring "ALLZ"
get FLAGS long
get SIZE long
get ZSIZE asize
clog "dump.dat" 0 ZSIZE SIZE

I didn't edit the original source code so it works exactly as expected

[chrrox]

It does work in his exe posted here.
https://github.com/Brolijah/Aqualead_LZSS/releases

[Brolijah]

It does work in his exe

This is half right... The file decompresses right, from what I see in the memory viewer. However, the destination pointer goes past the expected EOF... I'm looking into why that is now. In the code I shared, I checked on return if the DST equaled the EOF, and if it's wrong then I assume that to be the result of an error.
Truth be told, the game I reversed the decompression routine from didn't have any return checks validating it decompressed properly. In that game, it just exited when finished. (So my "error checking" is potentially flawed because I had to wing it.)

EDIT: I'm tired and jumping the gun. I just saw chrrox shared a different file than what I was given a short bit ago. The 33 file shared in this thread does decompress correctly in my tool. The one I was given (38) is the one with the weird dst error I just described.

EDIT2:

Honestly I guess I didn't test the code, not even sure if I had samples.

I gave like 15 sample files with the ALLZ code...

[aluigi]

Honestly I guess I didn't test the code, not even sure if I had samples.

I gave like 15 sample files with the ALLZ code...

Ah ok, so I tested it for sure.

Sorry, it's just that I didn't remember it, which is positive since it means it was very simple to implement and I had no bad memories about it

[aluigi]

This is how the code is called from quickbms:

Code: Select all

size = ALLZ_Decode(&out /*don't worry, doesn't get modified*/, in, zsize);

- out is the pre-allocated output buffer
- in is the input (from byte 0)
- zsize is the total size of file 33

It's all correct here.

[aluigi]

The current code on github has been updated since the one I implemented in quickbms and that may be the reason:

Code: Select all

<new vs old>
138c170
<                             *dst++ = *encoded_src++;
---
>                             *dst = *encoded_src;
271,272c303,306
<
<     return ((dst == decoded_eof) || (dst == (decoded_eof+1))) ? fullSize : 0;
---
>     // Eh, seems safe.
>     return ( ((encoded_src == encoded_eof) || ((encoded_src+1) == encoded_eof)) &&
>              ((        dst == decoded_eof) || ((        dst+1) == decoded_eof))
>            ) ? fullSize : 0;

[Brolijah]

Ahah... Apologizes for that. And apologizes in advanced again because I still need to fix one more thing with the decompression. I mentioned above a strange scenario where the dst jumps past the EOF, although the decompression was already completed. I'm waiting on a contact to finish testing the changes with a game which uses several ALLZ compressed files and inform me if it is 100% successful. I'm certain the solution will be one of two minor changes to the function.

Personally I'm a tad confused how I managed to finish the decompression "ahead" of the original assembly routines, but I won't complain if it works.

[aluigi]

Ok, keep us update when the new version is out

[Brolijah]

https://github.com/Brolijah/Aqualead_LZSS

Results are in, and I believe I've perfected it. The real fix turned out to be changing this:

Code: Select all

while((dst + disp_length) <= decoded_eof)

... to this:

Code: Select all

while((dst + disp_length) < decoded_eof)

I also have a few minor comments since I looked at how you implemented the functions. You don't need most the includes that I had for my tool. You don't even need "aqualead_types.h" since ALLZ_Decode doesn't contain the magic check. (In the games and my tool, that check is done outside the function before calling it.) The only includes that are actually needed for ALLZ_Decode are <string.h> and <stdlib.h> for these lines right here:

Code: Select all

if(!(*ptr_dst)) // stupid-proofing in case the caller didn't malloc a buffer
{
    *ptr_dst = malloc(fullSize);
    memset(*ptr_dst, 0, fullSize);
}

Other than that, and this one's just a tiny nitpick, I'd maybe suggest referring to the compression as ALLZSS for consistency with Aqualead's naming convention and so that others don't think its related with the LBALZSS formats that are also in quickbms. (Aqualead's names are always prefixed with AL, from their magic values to even their class names.)

[aluigi]

There is also that "if(encoded_src <= encoded_eof)" that should be fixed (not mandatory but better to do it).

Maybe at the end of the function you can just use:

Code: Select all

return dst - ptr_dst;

That would make the function more "dynamic".

[Brolijah]

From what I see in my pseudocode notes from both Ghidra and IDA, that line should be correct...

Code: Select all

// Ghidra
if (param_src <= pbVar6 + param_srcSize)
// IDA
if ( ptr_encoded_src <= &v3[arg_srcSize] )

I think you forgot a * and meant:

Code: Select all

return dst - *ptr_dst;

Even if I return the difference between the dst and it's starting point, that doesn't verify decoding didn't accidentally jump past the boundaries of its buffer. If the dst is beyond the expected EOF, that return wouldn't confirm it decompressed exactly as intended. (Except for the heap error that would probably occur.)

[kakarash]

HI, i wanna extract Urban Chaos PS2 games files (.000 .001 .002 and .wad files)

i have tried all scripts and tools but it wont work any one have an idea how to extract those files ?

[aluigi]

@kakarash
you are off-topic
And please read viewtopic.php?f=21&t=4 first.

[kakarash]

sry i'm new

[Brolijah]

While I was attempting (and failing) to build quickbms in Windows, I found this error with the swprintf macros which refuses to compile in against my GCC7 and GCC8 toolchains due to arithmetic on a pointer.

Code: Select all

// utils.c :  Line 439 in build_lpstrFilter()
for(i = 0; (f = g_filter_in_files[i]); i++) {
    myswprintf(fW, /*(ret_len - fW)*/ ret_len / sizeof(wchar_t), L"(%s)", native_utf8_to_unicode(f));
    fW += wcslen(fW) + 1;   // correct
    myswprintf(fW, /*(ret_len - fW)*/ ret_len / sizeof(wchar_t), L"%s", native_utf8_to_unicode(f));
    fW += wcslen(fW) + 1;   // correct
}

You make a comment about swprintf being borked in MinGW GCC7, but I can't find related error information on this particular function. Even if it was related... I really can't imagine why you subtract the pointer from the buffer length and use that as a size argument.

Also, before I make a thread on the subject: Do you have a specialized makefile for building in a Windows environment? Or do you only cross-compile? I've spent 3 days trying and consistently failing with errors that would only arise from compiling within Windows. (It's been a complete nightmare.)

[aluigi]

Indeed that math doesn't have sense since there is a "ret" missing there, I have no idea why gcc didn't yell at me.
The good news is that the function is NEVER used.

I use a .bat file for doing the whole compiling job on Windows using gcc 4:
http://aluigi.org/bms/compa.bat