QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[Shokoniraya]

so QuickBMS can open large files, so why it can not replace huge binary?

Solution:

Code: Select all

get FILE_SIZE asize
log "my_file.bin" OFFSET SIZE

1- use 64bit version of c functions
2- if my_file.bin size is bigger than SIZE, set file size to INCREASE_SIZE

Code: Select all

xmath INCREASE_SIZE = (FILE_SIZE + (my_file.bin_size - SIZE))

and overwrite my_file.bin from OFFSET location

3- if my_file.bin size is smaller than SIZE, set file size to DECREASE_SIZE

Code: Select all

xmath DECREASE_SIZE = (FILE_SIZE -  (SIZE - my_file.bin_size))

and overwrite my_file.bin from OFFSET location

but must moving forward or backward bytes of file and then doing overwrite

for compression, must create temp files in input file directory and overwrite it (if its not possible, then just do this for normal log and return error for clog)
but i think current way of QuickBMS is wrong to load data in memory

[aluigi]

@spiritovod
thanks for the feedback, the problem is caused by that modification I made in the beta regarding the usage of \ escapes in the printf 'p' setting of the String command.

Code: Select all

string NAME p "%s\%s" PATH NAME

If it screwed this script it would probably do it with others too.
Maybe it's better if I revert it to the original usage since there is no real advantage for users to have the escape feature there (which can be emulated by creating the format string with set VAR binary "format\\string").

@Shokoniraya
mah quickbms is ok right now, these operations look quite complicated and out from the scope of the tool.

[spiritovod]

Could you please check operations with "double" - apparently some of them are kind of broken in 4gb_files version in all available releases since its original implementation. For example, "get SOMETHING double" is producing "can't read 1 byte" error for any valid input - but usual quickbms version works fine. Also, I don't quite understand if it's possible to somehow cast or convert double value to integer with quickbms commands (without explicit calculations) - in the documentation it's suggested to use verbose operators from available in math (like ?lrint), but I couldn't get it to work as expected.
Update: I got it, usual quickbms version is limited to 2GB in any math or log operations due to signed output, since any number bigger than 0x7FFFFFFF is considered negative and can't be used properly in math, cycles, etc.

Here is valid double in little endian for tests: 00 00 80 8E 47 02 E0 41 (should be 2148678772 or 0x80123C74 in hex). Still curious why devs are using double for offsets, is not it platform dependent

[prefect]

Hi Luigi & everyone,

I don’t know if this is a bug or an intentional feature but I put it here.
Maybe I wrote a wrong script.

File "probain.txt"

Code: Select all

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

(any content, not particularly important)

BMS script "proba.bms"

Code: Select all

putvarchr MEMORY_FILE 5000 0        # allocate memory
log MEMORY_FILE 0 0                 # create the file
putvarchr MEMORY_FILE 0 0xAF        # put quote char
set SIZE 1
append                              # enables the append mode
set TOKEN_NAME_LENGTH 6
log MEMORY_FILE 3 TOKEN_NAME_LENGTH
math SIZE + TOKEN_NAME_LENGTH
putvarchr MEMORY_FILE SIZE 0xAE        # put quote char
math SIZE + 1
putvarchr MEMORY_FILE SIZE 0x0a0d short  # put CR and LF char
math SIZE + 2
savepos SIZE MEMORY_FILE
log "probaout.txt" 0 SIZE MEMORY_FILE
append
cleanexit

This is intended to extract 6 chars from probain.txt surrounded by special quote chars and followed by CRLF. The problem is that closing quote and CRLF is not written into probaout.txt. If the savepos line is commented out then they are written perfectly. It seems that MEMORY_FILE file position is not updated instantly (or at all) after the last PutVarChr.

[spiritovod]

@prefect: putvarchr doesn't change current position in a file (including memory ones), but savepos is saving current position, which is different from what you expect - in this case it would be at 7 after log operation with append enabled. Basically, those results are totally expected.

[prefect]

@spiritovod: Thank you for your reply. I found this in quickbms.txt:

Note that from QuickBMS 0.11 the Append command also affects the Put* commands (Put/PutDString/PutCT).

This implies some very recent changes about Put* code, maybe some behavioral changes occured. (Although PutVarChr is not mentioned here.)

A little bit unexpected by me that Log “knows” where the last PutVarChr left but SavePos doesn’t use that position.

It’s not a big problem, anyway. I can live with that. Or just not using SavePos right after PutVarChr but doing my own math as it is done in this example script.

[spiritovod]

@prefect: It seems you got it all wrong. Append command doesn't affect putvarchr - affected commands are explicitly stated: Put/PutDString/PutCT, and it makes sense due to how they handle position. Also, Log command doesn't write to current position in append 0 mode, but to the end of file. In you example you're allocating 5000 bytes for memory file, but after first putvarchr actual file will be one byte long, so Log is writing 6 bytes after it and current position will be set to 7, where it resides till the end, because you don't disable append mode. And putvarchr doesn't write to current position either, but to specified in the second argument offset. The rest was explained in my previous post.

You can check output from the following sample and try to change append mode to 1, in this case Log will write to current position in a file:

Code: Select all

set MEMORY_FILE binary "\x01\x02\x03\x04\x05"
set MEMORY_FILE2 binary "\x06\x07"

goto 3 MEMORY_FILE
append
   log MEMORY_FILE 0 2 MEMORY_FILE2
append
get SIZE asize MEMORY_FILE
goto 0 MEMORY_FILE
getdstring CHECK SIZE MEMORY_FILE
print "Mem file: %CHECK|hex%"

Also, here is your script reworked with Put command for comparison:

Code: Select all

putvarchr MEMORY_FILE 5000 0
log MEMORY_FILE 0 0
put 0xAF byte MEMORY_FILE

set TOKEN_NAME_LENGTH 6
append
   log MEMORY_FILE 3 TOKEN_NAME_LENGTH
append
put 0x0A0DAE threebyte MEMORY_FILE
get SIZE asize MEMORY_FILE
log "probaout.txt" 0 SIZE MEMORY_FILE
cleanexit

[prefect]

@spiritovod: Thank you again. Now I see my mistake clearly, somehow I happened to found PutVarChr first (the one Put* which uniquely doesn’t change file position), instead of Put/PutDString/PutCT which I should have used, because it does exactly what I needed.

BTW thanks for the reworked script and the example, nice coding.

[Sruffy]

I know its been very long but I couldnt find a way to make a new post and ask for a posisble solution. I mod Tekken 7 and while trying to unpack a mod file, this error happens. Tekken 7 has a specific script written by someone which I use but its not working on some of mod files. Can someone please help?

[Sruffy]

This is the script I use for normal unpacking

[Sruffy]

And this is the mod file I am trying to unpack
https://www.mediafire.com/file/zzeh2cas ... P.pak/file

[Sruffy]

This problems is not so uncommon.
Many libraries that compress using the deflate algorithm (known as zlib if there is a header and a crc before and after it) sometimes are not able to match the ratio or other libraries in some conditions.

For example zlib is not so good and for that def_item_trophies.xml file you provided you get 1996 bytes.
7zip compresses it to 1984 bytes.
The uberflate/kzip solution reaches 1970 bytes.

A good candidate is zopfli but it's unable to offer good results with various types of files.
In this case it's good and I can reach 1939 bytes so I will for sure add it to the next version of quickbms (I already evaluated it months ago but I rejected it just due to its results) but I guess I have to write a sort of "options fuzzer" for finding those with the best results.

[Sruffy]

Hi everyone, I mod Tekken 7 and while trying to unpack a mod file, this error happen. Script I use is attached below for reference. Could someone please help?

Error: the compressed zlib/deflate input is wrong or incomplete (-3)
Info: algorithm 1
offset 00000049
input size 0x0000055e 1374
output size 0x00000e29 3625
result 0xffffffff -1

Error: the uncompressed data (-1) is bigger than the allocated buffer (3625)

[3IMiner]

Hello,aluigi,I think this is a script problem...
viewtopic.php?f=9&t=17141&p=72570#p72570

[aluigi]

@Sruffy and 3IMiner
You are off-topic

@spiritovod
ok I fixed that.

I just released quickbms 0.12 as stable version, it's the beta with the fix for the float bug plus a couple of simple patches for compilation on other platforms and updates libraries (like zlib and lzma).

I decided to go with the stable version because the beta didn't give any problems for so many months and I have no plans to work further on it.

P.S.: yeah I know that the changelog sucks this time, for any details it's better to check what I wrote in my previous posts about the beta.

[johnz1]

Hello

I just upgraded QuickBMS from 0.10.1 to 0.11. I'm running into a problem with reimport (the original, not reimport2 or reimport3). 9 out of 743 files that used to be able to be reimported are now failing because the compressed size is too big. If I switch back to 0.10.1, they all can be reimported without an error. I'm using "nbajamfire.bms" 0.2.3a with QuickBMS to reimport texture files.

I don't see anything in the 0.11 release notes about changing the compression algorithms. I've also noticed that the reimport in 0.11 is much faster than 0.10.1 - which is nice!

Thanks

I'm happy to report that this is fixed in QuickBMS 0.12. Thank you!

[spiritovod]

aluigi: Thanks for fixing doubles. I only checked it with "get" so far, but probably "put" and reimport also should work fine now.

Not sure if it's supposed behavior or just missed point in documentation, but one must consider actual internal order for multi-dimensional arrays and address then accordingly. For example:
putarray 0 FILES 0 0
putarray 1 FILES 0 0
^ will not work, because it will be putarray 0 / putarray 1 / putarray 1 / putarray 2 and they will overlap.
Correct usage would be:
putarray 0 FILES 0 0
putarray 2 FILES 0 0

[Jozaca1199]

Hello, I have a doubt about using quickbms with the purpouse of modding a game. Does the "NOT games" in the title mean that I should post this in another thread?

[sparr]

Code: Select all

string NAME = ".foo"
log NAME 0 0

This script produces a zero byte file named "foo"

If I run `quickbms -f ".foo"` then the file is extracted, still with the name "foo"

If I run `quickbms -f "foo"` then no file is extracted.

Why is the leading dot stripped? Since it's stripped, why does the -f filter still match the unstripped name? Is this a bug, possibly related to file extension handling? How can I avoid this?

EDIT: The test case above is meant purely for illustrative purposes. My actual use case is a real archive containing files with names starting with a dot, which get extracted without the dot.

[Jozaca1199]

Hello, I have a doubt about using quickbms with the purpouse of modding a game. Does the "NOT games" in the title mean that I should post this in another thread?

I still have the same question