QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[aluigi]

@z4ruz
Regarding "2) Print array elements as hex issues" where you get "1" and 31 instead of 1 and 01.
The reason is that the variables arrays (subvar or whatever we call them) only store a string representation of the variable.

@johnz1
The reason why in 0.10.1 reimport.bat worked while it fails on 0.11 is that I slightly changed the behavior of the zopfli compression.
In short in 0.10.1 it was a more extreme and slower implementation.
Anyway I'm going to "fix" it by reenabling the extreme mode with reimport1

@spiritovod
Ah, remember to use ever {} instead of * in the filters because Windows is going to interpret that char before quickbms and doing a mess

[aluigi]

@spiritovod
The access violation you noticed is probably the same bug reported in ttgames few posts above.
I can't replicate these issues but I guess it's related to the impossibility of allocating the memory, even if I don't get why 0.10 worked.
Did you use the -9 option maybe?
We need a simple way to replicate the issue.

[aluigi]

@z4ruz
Regarding "string var2 r var" use "string var2 0r var" and it will correctly handle the NUL byte

[aluigi]

The following is a beta that should fix most of the reported issues:
http://aluigi.org/beta/quickbms_exe.zip

(If interested in the source code, the diff from 0.11 is on http://aluigi.org/beta/quickbms.diff, plus I updated some compressions like lzma, zstd and blosc)

[gamelandresearch]

i don't know what to say, words not enough...
i can't decrypt .pak still i got error after updating quickbms to latest
that's problem decrypting .pak:

-------------------
*EXCEPTION HANDLER*
-------------------
An error or crash occurred:

*EH* ExceptionCode c0000005 access violation
*EH* ExceptionFlags 00000000
*EH* ExceptionAddress 0AA15B90
*EH* NumberParameters 00000002
*EH* 00000000
*EH* 381A0416

Last script line before the error or that produced the error:
46 calldll MEMORY_FILE10 decryptPackage tcc RET MEMORY_FILE SIZE
coverage file 0 100% 2201972 2201972 . offset 00000000
coverage file -1 0% 0 2201972 . offset 00000000
coverage file -10 0% 0 1018 . offset 00000000

i used script: https://zenhax.com/viewtopic.php?f=11&t=15965

[aluigi]

@gamelandresearch
You are talking about a different topic.
I can't help with that because that function (that comes from an action script here https://github.com/LolHacksRule/AngryBi ... lLoader.as) doesn't have any sense to me (mRandom declared as 32bit signed integer but with unsigned right shift of 35 and division of 31 bits at the end).
The crash is caused by the second cycle, even if you limit it the result will still be garbage.

[spiritovod]

aluigi, sorry for the late reply, I was quite busy recently. I've checked all 5 games with azure package format (the one which is used in Dragon Raja as well) and couldn't reproduce that memory allocation error, maybe it was an unfortunate coincidence for some particular version of one of them.

And I'm aware about {} usage, but it was more about difference between 0.10.1 and 0.11 behavior - if the same parameters didn't work for both of them, it would be totally understandable, but in 0.10.1 it works just fine. Maybe you need to adjust documentation on this matter, because currently * is used as wildcard in most samples in documentation.

[aluigi]

Do you mean the bug with -F ?
As far as I know I fixed it in the beta, the * got expanded by windows on both 0.10.1 and 0.11 but on the latter it had hit a new bug.
Definitely going to replace * with {} in most of the examples.

[spiritovod]

Yes, I mean -F usage in batch scripts. Also, sample from this post now produces "the input folder is empty" error with latest beta (as said above, it works fine with either 0.10.1 or {} as wildcard in the script).

[tbmq008]

i have a Windows 10 PC with 6GB of RAM and it uses more than half of that and i cannot decompress a big 1.53GB file.
memory errors and all that.

Code: Select all

E:\quickbms>quickbms "E:\[REDACTED]\xcompress_file.bms" "E:\[REDACTED]\Gen_Common.lin.bf" "E:\[REDACTED]\Gen_Common.lin.bf"

QuickBMS generic files extractor and reimporter 0.11.0
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Apr  5 2021 - 13:56:34)

                          quickbms.com  Homepage
                            zenhax.com  ZenHAX Forum
                     @zenhax @quickbms  Twitter & Scripts

- open input file E:\Microsoft - Xbox 360\MotionSports [RF][Kinect]_2\Data\CBin\Gen_Common.lin.bf
- open script E:\Microsoft - Xbox 360\xcompress_file.bms
- set output folder E:\Microsoft - Xbox 360\MotionSports [RF][Kinect]_3\Data\CBin\Gen_Common.lin.bf
- the folder doesn't exist, do you want to create it (y/N)?:
  y

  offset   filesize   filename
--------------------------------------
  00000000 1649442550 Gen_Common.lin_unpack.bf

- error in src\extra\xalloc.c line 618: xdbg_malloc()

Error: memory allocation problem
       No h recursos de memria disponveis suficientes para processar este comando.


press ENTER to quit

and yes this does kinda affect quickbms in a way since i had no idea xmemlzx "allocates" bigger sizes than necessary given how LZX compression algorithm actually works.
i'll be uploading a small sampler of that file upon request.

[aluigi]

@tbmq008
Unfortunately I have no solution for that problem at the moment.
Quickbms is compiled as 32bit application for compatibility reasons (in short it must work on any Windows version on any hardware) and therefore the memory allocation is limited to 2 or 3 Gb.
Since all the compression algorithms are buffer-to-buffer, it must allocate memory for the source and for the destination, so if the input is 1.5 Gb it probably needs to allocated 3.5 or more Gb.

No plans for releasing 64bit versions. Maybe I can think about doing that for quickbms_4gb_files.exe but I would like to get more feedback about this possible choice.

[tbmq008]

same error with quickbms_4gb_files, unfortunately. but whatever.

is "dynamic memory allocation" an OK thing to suggest? this would be particularly useful for decompression algorithms like xmemzlx given what i found.

is there any way to simply work around this issue? like having xmemlzx decompress raw compressed data only instead of reading everything.

i had to pull off massive hacks into one of my scripts which covered a file format that used this compression algorithm by having the script feed fake header data plus the compressed chunk data into memory so it can be decompressed into memory.

[aluigi]

@spiritovod
The reason of that error is that quickbms accepts a text file or a string with the -F option, because somtimes it may be useful to have a list of files in a text file rather than having a long command-line option.
My mistake is that probably I should limit the file input to the .txt extension (so -F list.txt is ok while -F list.dat is considered a string).
Let me know if that may be better and I will add it.
*edit* anyway there is something that I have to check in the beta because there was currently a NUL-byte check on the filter file that seems to fail
*edit2* yeah it's a bug, long story short the input files have the local folder in them so I have to find a simple solution: .\EnderLilies-WindowsNoEditor.pak

@tbmq008
quickbms_4gb_files is 32bit too, what I mentioned is a "possible" idea for the future.
Anyway I released a new beta of quickbms with a possible solution and a work-around.
The solution was already available in quickbms but only activated when using the -9 option, while now in the beta it's ever working. It consists in freeing the current buffer if reallocation fails (because realloc takes double memory).
The work-around instead is allocating only one buffer containing both input and output, obviously this is WRONG but it may work often since the input buffer is located at the end of the output one and the decompression should advance both input and output without overlapping.
A warning message is displayed when the work-around is activated.
beta exe:
http://aluigi.org/beta/quickbms_exe.zip

(source code diff is ever at http://aluigi.org/beta/quickbms.diff)

[aluigi]

@spiritovod
ok I just updated the beta and now it's all fixed and I even added support for input list encoded with UTF16 (UTF8 BOM, UCS2 BE/LE BOM)

[tbmq008]

alright, i'll try using quickbms 0.11.0 with the -9 option first then the beta.
i'll edit the post when i'm done.

---

tried the format and it immediately told me that there wasn't enough memory to decompress that one file.
tried the latter and... honestly, it wasn't too bad (works as intended) but i haven't seen quickbms "dump" the decompressed file yet so i used HxD to see what was going on there (HxD can open the "main memory" where there are various "processes" to choose from) and it seems that quickbms is still decompressing the whole thing in memory?

again, i used the xcompress_file.bms script with a 1.5GB compressed file. this is the first quickbms instance i've opened.
so i opened up another quickbms instance and used that same script with a 31MB compressed file and the file was decompressed just as expected. the output file was dumped just as quickly, which is within a second on my PC.
of course, how much is the file actually "dumped" into a decompressed one depends on how big the compressed file is in size to begin with i didn't expect the former to hang for so long.
i'll keep this updated whenever possible.

[aluigi]

I use the following script for generating a 1.5 Gb file in memory and decompressing it as an 1.8Gb with lzss and it "works".
I mean that there is no endless loop and it takes 6 seconds to do the job:

Code: Select all

math ZSIZE = 1500
xmath SIZE "ZSIZE + 300"
xmath ZSIZE "ZSIZE * 1024 * 1024"
xmath SIZE  "SIZE * 1024 * 1024"
log MEMORY_FILE 0 0
putvarchr MEMORY_FILE ZSIZE 0
goto 0 MEMORY_FILE
comtype lzss0
clog "dump.dat" 0 ZSIZE SIZE MEMORY_FILE

The endless loop happening there may be caused by overlapped input and output data.
The pointer in the destination may reach the pointer in the source since it's the same buffer, that may happen as written in the Alert message.
Or maybe it's just the decompression which is slow or the xcompress library which is unable to uncompress a so big amount of data, there are better chances that this is the real explanation.

I like this work-around and it may help in those very rare cases where such big files need to be decompressed.

In your specific case I suppose that on x360 the library handles the file as a compressed filesystem or similar, being able to seek and read parts of it without performing any decompression of the whole file.

[tbmq008]

interesting...
so, that script you've just made... it took like 2-3 minutes to "do the job" and even then you actually got the file. and a decompressed one at that, given that the compressed input is all generated into memory and all that.
i couldn't get anything similar with that one file despite being compressed into an LZX container.

[aluigi]

Do you get 2-3 minutes with the testing script above or with xcompress_file.bms?

[tbmq008]

testing the script above.
i had to use it with a text file.

[aluigi]

It's strange, here only 6 seconds and there all that time?

It's not really important since the goal of the test was only the memory allocation, but still yours is 25x slower.