QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[aluigi]

quickbms already uses the latest versions of the compression libraries.
It also already uses the best compression levels, example for lzo1x:
lzo1x_999_compress(in, insz, out, &len, wrkmem);

Using a better compression level doesn't mean that the output will be ever smaller, sometimes it can be bigger with some samples.

zlib is slow for various reasons.
Differently than other packing tools, QuickBMS is a reimporter and so its priority is the final size of the data which means using maximum compression for fitting the original size.
It even uses 3 different types of zlib if the final data is bigger than the original which means a huge amount of time.

I guess that you have that problem only with reimport.bat since reimport2.bat should just use the normal compression.

Searching binary data with quickbms is difficult because the tool is not meant for that.
I can try to improve findloc binary but I doubt in any better result.

What is that "getbinary"???
getdstring already does the job

[Shokoniraya]

getdstring can't do that job

here a clear example

Code: Select all

set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
get MEM_SIZE asize MEMORY_FILE

string HEX_STRING = ""
for i = 0 < MEM_SIZE
get BYTE_NUM byte MEMORY_FILE #same job as getvarchr to get every VAR in for-next
string HEX_BYTE p "%02x" BYTE_NUM #so it will make a real hex-string
string HEX_STRING + HEX_BYTE
next i
print "hex string: %HEX_STRING%"
string HEX_STRING h HEX_STRING #and it will make it byte, but its string! so \x00 will be a problem!

set MEMORY_FILE5 binary "\x00\x01\x00\x02\x00\x03"
findloc RESULT binary HEX_STRING MEMORY_FILE5 ""
print "Result: %RESULT%"

and can you do something to have some options to choose temp files for exe executing? some codec tools don't have [output] (example: program.exe -c [input] [output]), so output will have a random name

[aluigi]

That's exactly the job of getdstring.
What you are showing is something different related to strings (strings are NUL delimited) and findloc which is still related to strings.
I can do something for Findloc and maybe a work-around for String when used with some operators.

This is a limitation that affects me too and I want to find a solution, but it's NOT related to getdstring

Regarding the other request I can't do anything because quickbms gets the result from the output file generated by the program.
quickbms doesn't act as hooker or hypervisor of the program, it just creates the input file, pass it to the program and read the generated output file.

[Shokoniraya]

and about temp names, temp files created in quickbms temp folder, if there was a way to have last temp file adress that created in folder, then problem will solve

[Shokoniraya]

That's exactly the job of getdstring.
What you are showing is something different related to strings (strings are NUL delimited) and findloc which is still related to strings.
I can do something for Findloc and maybe a work-around for String when used with some operators.

This is a limitation that affects me too and I want to find a solution, but it's NOT related to getdstring

Regarding the other request I can't do anything because quickbms gets the result from the output file generated by the program.
quickbms doesn't act as hooker or hypervisor of the program, it just creates the input file, pass it to the program and read the generated output file.

and about getdstirng, that's right, that little example wasn't related to getdstring, but i'm saying that string will skip in \x00 (null delimited)
so if we use getdstring TEXT 15 on \x68\x65\x72\x65\x00\x61\x6E\x00\x65\x78\x61\x6D\x70\x6C\x65, then it will stop on \x00, but we need whole string, that's why im saying that there is a getbinary needed

[kotn3l]

Trying to extract PSARC files using the latest quickBMS and brink.bms versions from a new-ish PS4 game. I had to use the 4GB exe version, because the PSARC file I wanted to unpack is over 7GB.

Code: Select all

- error in src\extra\xalloc.c line 618: xdbg_malloc()
Error: memory allocation problem
       Not enough memory resources are available to process this command.

It can extract some psarc files pretty easily. I guess the problem here is that the file it wants to unpack is way too big. Any workaround? Or do I just wait for a fix? Is there a way to continue unpacking after that point?

[Shokoniraya]

there is two problem that i have found in QuickBMS

1) bms_folder
BMS_FOLDER can't get script folder as well
my_script.bms is located in "G:\my_scripts\works\pak_format\my_script.bms"

and this is my command
"quickbms.exe" -o "G:\my_scripts\works\pak_format\my_script.bms" "my_file.bin" "output_folder"
result is "G:\my_scripts\works\pak_format" and it will work

but if i use it without a full name and drive and file name, result is "G:\my_scripts\works" for this command
"quickbms.exe" -o "my_script.bms" "my_file.bin" "output_folder"

i think QuickBMS removing last sub-folder or sub-name from bms adress
problem is that i can't use it if i write script name without full path


2) EXECUTE command
EXECUTE in comtype command has a problem, and maybe it can't works well with Temp folder on a windows user name that has space character in it (like my system user name)
because i get this error
@calc\calc.exe INPUT_NUMBER / 32 >C:\Users\************\AppData\Local\Temp\quickbms_00000da80000000181adcaa4289aa2b5.tmp
The system cannot find the path specified.

here a example: http://www.mediafire.com/file/ew70myc2t ... m.zip/file

[aluigi]

and about temp names, temp files created in quickbms temp folder, if there was a way to have last temp file adress that created in folder, then problem will solve

There are tons of processes creating files there

Trying to extract PSARC files using the latest quickBMS and brink.bms versions from a new-ish PS4 game. I had to use the 4GB exe version, because the PSARC file I wanted to unpack is over 7GB.

That's off-topic here.

so if we use getdstring TEXT 15 on \x68\x65\x72\x65\x00\x61\x6E\x00\x65\x78\x61\x6D\x70\x6C\x65, then it will stop on \x00, but we need whole string, that's why im saying that there is a getbinary needed

I'm working on the next version of quickbms just in these days and the results are interesting because the String command is now no longer limited to NUL strings.
Obviously there will be some incompatibilities with some scripts and some headaches in some situations, I will upload a beta quickbms.exe soon so you can try the new features.

1) bms_folder
BMS_FOLDER can't get script folder as well

What's the current folder from where are you launching quickbms?
If I'm in g:\my_scripts\works\pak_format both the examples create output_folder where expected.

2) EXECUTE command
EXECUTE in comtype command has a problem, and maybe it can't works well with Temp folder on a windows user name that has space character in it (like my system user name)
because i get this error

Good, this is definitely something to fix (and apparently related to the name(blah).exe bug)

[Shokoniraya]

about temp folder, yes, there was so many files on there, i'm not talking about last file that created in temp folder (based on time stamp), i mean last file that EXECUTE created it
quickbms dumping that file on temp folder, right? so it will be useful if get dumped file path from dumper function

and about bms_folder, i upload a example, please use it under two sub folder, like C:\folder_1\folder_2\bms_folder
http://www.mediafire.com/file/2zf956e2j ... r.zip/file

in just_script_name.bat, last sub folder is missing from result

and can you add LZX compressor too?

using -n in QuickBMS, is there any way to get file name in a download link that does'ny have any name?
becuase some links does not have any name or not a normal name like php file link
for examles, these links
"https://zenhax.com/download/file.php?avatar=2_1542023899.png"
"https://zenhax.com/download/file.php?id=9002"
"https://avatars1.githubusercontent.com/u/5502283?s=460&u=861e923455e5411cd3bfb07a8c533db777ea9276&v=4"

maybe they are using some java script method or redirected links
and i thing downloading a file from a server with QuickBMS is slow (downloading speed)

[aluigi]

I still miss the purpose behind the name of the temporary files created by quickbms.
They are immediately deleted after the launched executable terminates and the content of the output file is read by quickbms.

I found and fixed the bug of BMS_FOLDER.

LZX is already available since years and in many variants.
Comtype:
- lzx, the version used in an old tool
- xmemdecompress, the most used in games
- lzxcab, used in the Microsoft CAB files (there is also a _delta version)
- lzxchm, used in the Microsoft CHM files (there is also a _delta version)
- cabextract, same as lzxcab I guess
- DS_LZX

The compressor is already available for xmemdecompress and ds_lzx.

Regarding the last question it's a bit complicated.
The library used by quickbms doesn't return the original name and even in that case I would have to choose between the original URL and the name of the file provided by the server.
Since quickbms is NOT a downloader and it only needs the content of the file, it's more appropriate to keep the original URL.
What I will do in the next version is improving the available names, example:

Code: Select all

filename     file.php
fullname     https://zenhax.com/download/file.php?id=9002
basename     file
extension    php
fullbasename download/file
filepath     download

quickbms has the same download speed of anything else.
Probably you are comparing it with downloaders downloading the file with multiple connections which is totally unrelated.

[Shokoniraya]

sir aluigi, one other thing

can you update QuickBMS guess function for file extension? like mp3 files, or adding more guess for media files

[aluigi]

MP3 files don't have an header since they are just a sequence of compressed frames without any "magic" value (often they start with byte 0xff but it's useless).
Some years after mp3 has been made it started to be used the ID3 struct for containing metadata like artist, name of the song and so on.
QuickBMS uses the ID3 "magic" for guessing the mp3 extension, so there is no extension if there is no ID3.

As far as I know quickbms already supports most of the format available, if something is not supported it's just because there is not "magic" to guess.

Let me know if you have suggestions about a specific format

[Shokoniraya]

found a problem
QuickBMS asks -w in slog exporting (tested with last beta)

here a script
viewtopic.php?f=12&t=14240&p=59519#p59519

i've attached a sample file too (in this comment)

[aluigi]

Fixed

[Shokoniraya]

problem solved (now, im sure UltraEdit broked my file, gotta leave it)

[khachai]

I got an error like this, please help fix it
https://i.imgur.com/QEh6WDa.jpg

[Shokoniraya]

sir aluigi, in tested QuickBMS in these days and i can say it's working great

but there is one problem, deflate
zlib/deflate decompression works fine, but looks like some games and even some programs can't read deflate or zlib files that compressed with QuickBMS
such as limbo game, the evil within 1, 7z Gui and some other games that using zlib or deflate
also, if you can use same version of deflate that used in winrar in QuickBMS, it could be great, i compressed files with winrar (in zip mode) and used it and worked fine in those games
(sorry for my bad grammar)

can you replace current version of deflate library of QuickBMS with a safe version one or a public one? i think you should use this: https://zlib.net/zlib1211.zip

[aluigi]

Wait wait wait.
What you mean with winrar?

winrar creates zip and rar archives.
quickbms extracts data and can't create archives, it can only reinject compressed data back in the original archive.
Obviously a zip after reimport mode will never work due to the different crc.

the_evil_within.bms is a different case because it's not a real deflate, it's customized and indeed the only way to dump that data was using deflate_noerror that ignores errors.

P.S.: quickbms uses only zlib in reimport2.bat, the zlib->zopfli->kzip sequence is only used in reimport.bat for fitting in the original space.

[Shokoniraya]

i compressed raw file with winrar, and then, i extract compressed part of zip file that created with winrar (in PK) with hex editor by hand, and used it in those games and works fine, but compressed raw file that QuickBMS creates not working on some games

and in other test
i compressed a .txt file file with winrar to zip archive (with deflate 0x08 flag), now i have a normal compressed zip file
and then, i compressed same .txt file with QuickBMS to deflate and replaced it with compressed part of zip file with hex editor, and i updated zip details with hand too, such as crc32, size or compressed size and offsets (pk offset and table offset at the end of file) and it fails in extraction!

and i repeat same job, but in this time, i used 7z to compress deflate except QuickBMS (creating a zip file in 7z Gui) and after doing same job, it worked with no fail!

i did same thing on the evil within with 7z compressed deflate raw file (again, compressed raw file, not archive! because i cut compressed part of zip file by hand and pasted it to my raw file) and worked!
also, i repeated same job for the evil within with winrar compressed raw file and works file
but when i used deflate of QuickBMS, game crashed!
so if it;s a customized deflate, so why winrar and 7z compressed files worked fine?
QuickBMS deflate and zlib should be replaced with a correct one

i think it was a miss understanding in your reply, because i'm talking about compressed file, not archive, i just used winrar to get compressed raw file

[aluigi]

Ah cool

So you can do the same test with quickbms too using the following 3 scripts:

Code: Select all

comtype kzip_deflate_compress
get SIZE asize
clog "kzip.dat" 0 SIZE SIZE

Code: Select all

comtype ZOPFLI_DEFLATE_COMPRESS
get SIZE asize
clog "zopfli.dat" 0 SIZE SIZE

Code: Select all

comtype deflate_compress
get SIZE asize
clog "zlib.dat" 0 SIZE SIZE

Please let me know what of the created dumps does work and NOT work.

Ah, just a note, I found that quickbms uses advance_comp for zlib and deflate, not zlb because it gives better ratio.