QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[spiritovod]

@Shokoniraya: This script doesn't make much sense to me. Usual construction will look like this (aside from that you're trying to add the same file to itself or something):
append -1
for ...
log ...
next i
append

Also, consider using MEMORY_FILE or TEMPORARY_FILE before writing actual file if possible. Otherwise you may be stuck with core logic of how quickbms is working with files. You may take a look at existing scripts using "append" for better understanding.
Anyway, proper example script and expected results would be nice.

[Shokoniraya]

append - 1 should add binary to strat of file but it has problem

and i can't dump a huge file in memory file (my system is low end), however, test it with a small file (40-byte) and you will see that binary will overwrite (not overwrite actually, a kind of problem in file enlarging)

[spiritovod]

Now I see what you mean, "append -1" only works for a single iteration, because it's allocating additional memory only once - on the 2nd iteration and the rest it stop allocating and start to rewrite file (like in "append 1" mode). Not sure, but it may be actually related to position checks for memfile. If you allocate required memory manually, it works fine though.

I hope aluigi will take a look at that when he has time. Also, some basic example would be appreciated, since I'm also confused with how "append 1" works - in documentation it's stated that you can choose position where to write stuff in rewrite mode with goto, but you expect that goto will work on current file. You may expect that you should open result file as 1, then something like "goto OFFSET 1" and then "log ..." into that file from memfile, but that doesn't work as expected.

[Shokoniraya]

there is two problem that i've found

1) codepage problem
when codepage used in a script, some empty folders will be create with none-codepaged names, but it just a junk folder and codepage works correctly

2) unicode names for input file
it's clear that cmd can't support unicode as well, but for %%x can be used in any file, no matter file names has space character or unicode names and quickbms can't support unicode in input name

[aluigi]

when i want to run a exe with a none-standard cstring name like ( or ) that should used " in beging and end of name, in execute command in bms script, command not working fine

"zlib(v2.4).exe -c #INPUT# #OUTPUT#"
or even this one: "\"zlib(v2.4).exe\" -c #INPUT# #OUTPUT#"
or this: ""zlib(v2.4).exe" -c #INPUT# #OUTPUT#"

The problem is not in the string, I guess it's quickbms that uses parenthesis as escape so using " or \( doesn't solve it.

I'm quite sure there is a reason behind that behavior but I will check it when I will work on the next release

(I will slowly try to answer to all the posts, just wait)

[aluigi]

sir aluigi

there is a problem in QuickBMS that always making some problem in text importing
there is some string like \n or \f or \e
but can you skip none cstring error? like \:

Error: cstring() failure, your input string has some wrong escape sequences or
it's not a valid escaped string

i know i should use \\:
but \: not used in cstring, right? becuase there is not any \: or \k too, so error and stop the process can't helping, then simply ignore unavailable escapes

and i know if i export that text again, it will turned to \\:
but there is no problem with that, so can you please ignore wrong escapes in next version?

thank you

Are you using strings with invalid escapes? And why?
Are you talking about the Slog command?

I probably need a complete example for understanding if it's a problem or not.

The error is necessary because people need to know if something failed.

[aluigi]

sir aluigi, about cstring errors, i know there is some problems for that if you ignore it as default, but it's better to make a command option to ignore incorrect \* cstring

and about unicode converting that skips at zero, normaly if we want to read a 64 bytes that stored a name in it, it will stop at zero (null). so there must be a option to ignore zero bytes too
because of such cases

Code: Select all

63 6F 6D 6D 6F 6E 5C 00 00 00 00 00 75 69 5C 00 common\.....ui\.
00 00 00 00 00 00 00 74 65 78 74 00 00 00 75 72 .......text...ur
00 00 65 00 00 00 5C 00 00 00 66 6E 74 5F 00 00 ..e...\...fnt_..
00 00 00 30 2E 00 00 00 00 00 66 6E 74 00 00 00 ...0......fnt...

I don't know what you mean.
And that's not unicode.
Clearly a format like splitted strings for manual reassembling in different ways and not a string.

[aluigi]

another problem, but i don't know that if it's a QuickBMS feature or not

append -1 not working as well, i think it should add binary to start of file but has problem

please test it on a 40 byte file (you can see it better in a small file)

Code: Select all

get SIZER asize

for i = 0 < 5
append -1
log FILE_APPEND 0 SIZER
append -1
next i

Ok, here I guess it's a bug because I tried 10 Log of 3 bytes each but the output is 6 bytes instead of 30.
Added to the list of things to check.

Ah, it works perfectly with MEMORY_FILE so it's only related to log to file.

Likely "append -1" is never used

[aluigi]

Now I see what you mean, "append -1" only works for a single iteration, because it's allocating additional memory only once - on the 2nd iteration and the rest it stop allocating and start to rewrite file (like in "append 1" mode). Not sure, but it may be actually related to position checks for memfile. If you allocate required memory manually, it works fine though.

I hope aluigi will take a look at that when he has time. Also, some basic example would be appreciated, since I'm also confused with how "append 1" works - in documentation it's stated that you can choose position where to write stuff in rewrite mode with goto, but you expect that goto will work on current file. You may expect that you should open result file as 1, then something like "goto OFFSET 1" and then "log ..." into that file from memfile, but that doesn't work as expected.

Append 1 is meant to be used on MEMORY_FILEs because you can't move on output files on disk.

The following scripts use "append 1" if interested:
http://aluigi.org/bms/bsc.bms
http://aluigi.org/bms/clutch.bms
http://aluigi.org/bms/gh_pak_chnk.bms
http://aluigi.org/bms/marvel_ultimate_alliance_2.bms (there is also a comment explaining why it gets used)
http://aluigi.org/bms/ravioli_reimporter.bms
http://aluigi.org/bms/spiderman_pcpack_nch.bms
http://aluigi.org/bms/the_order_1886.bms
http://aluigi.org/bms/zombiu.bms

Regarding the script of Shokoniraya, it's necessary to use "append" without any argument for resetting the append mode, so "append -1" + log + "append".

[aluigi]

there is two problem that i've found

1) codepage problem
when codepage used in a script, some empty folders will be create with none-codepaged names, but it just a junk folder and codepage works correctly

2) unicode names for input file
it's clear that cmd can't support unicode as well, but for %%x can be used in any file, no matter file names has space character or unicode names and quickbms can't support unicode in input name

I will check what causes that "Ianei" folder to be created.

Regarding the second problem I can check why the name isn't correctly handled as utf8 when quickbms points to the folder, probably a bug in the function that scans the folders.

[spiritovod]

^ Thanks for those examples, now I see how "append 1" should be used in different situations. Also not sure, if "append 1 -- -- append 1" is correct construction in bsc script, can you please check it?
As for the "append -1" problem, if you'll use it just once (append -1 / log to file / append) it will work as expected, so it looks more like positioning/allocating issue for multiple iterations.

[aluigi]

"append 1 -- append" is the correct one.

[Shokoniraya]

hello sir aluigi
can you check these two things?

1) compression library
QuickBMS zlib compression (in reimporting) is really slower than any other zlib compressor tools
can you add a heavy optimized zlib in QuickBMS?

and some compressed lzo1x files are bigger than original files, (i mean recompressing original files without any change), i think you should change it with a newer version
about LZX compression, i can see QuickBMS already has decompressor for that, can you add LZX compressor too?

2) search content of MEMORY_FILE in findloc
i want to read a few byte (14 kb) and use it in findloc, is this possible?
after reading QuickBMS txt file, i could not find any useful thing to do it
can you tell me its possible or not, or how can i do it? becuase its really needed

[spiritovod]

^
1) How can you compare compression tools with quickbms if reimport function is doing more work than any generic compressor by default? It's not about zlib, it's about reimport feature in general.

2) You can do it with something like:
getdstring SEARCH BYTES_LENGTH
# not sure if "set SEARCH binary SEARCH" conversion is required
findloc RESULT binary SEARCH MEMORY_FILE ""

[Shokoniraya]

thank you spiritovod for your answer
but QuickBMS zlib and lzo1x is not optimized, you can compress files even without using reimport option and just with adding _compress to your comtype, if QuickBMS have compressor for that comtype

Code: Select all

comtype zlib_compress
get FILE_SIZE asize
clog compressed_file.zlib 0 FILE_SIZE FILE_SIZE

and getdstring is not a good option for reading a few byte, its for null terminated string and that means output result will stop on first null byte (zero \x00"), so can't use it to read \x00\x01\x00\x04 and search it, getdstring is works just like get TEXT string, but just can adjust number of bytes to read it
and i tried MEMORY_FILE and not working

[spiritovod]

The problem is actually not that. I remember that aluigi said somewhere that it's not possible to directly use a variable to search in binary file like that, but it's rather because you should understand how quickbms is working with strings / cstrings / numbers and convert them (for example, I don't). And it seems there is a problem with leading zeroes, which works as delimiters as well. So, for example, this code will work as intended:

Code: Select all

set SEARCH binary "\x11\x00\x22"
set MEMORY_FILE binary "\x01\x01\x02\x02\x00\x11\x00\x22"
findloc RESULT binary SEARCH MEMORY_FILE ""
print "Result is: %RESULT%"

but the following ones doesn't (considering that you change memfile accordingly):
set SEARCH binary "\x00\x11\x00\x22"
set SEARCH binary "\x01\x00\x02"

The example above works only because it was created this way. Actually it's searching only "\x11" in the memfile, because SEARCH is getting trimmed due to the delimiter (which is obviously non-intented in quickbms).

----------------------------------------------

And getdstring works in a different way, it's about representation as well. Take a look at the example:

Code: Select all

set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
getdstring SEARCH 4 MEMORY_FILE
string SEARCH b SEARCH
string SEARCH - "0000"
string SEARCH >> 2
print "Search: %SEARCH%"

I believe if leading zeroes problem will be fixed somehow, you can find anything with getdstring and some workaround regardless of zero bytes delimiters.

[Shokoniraya]

I know how it works and thank you so much to spending your time to explain it

so one of things that we need in next version is getBinary

Code: Select all

getbinary MY_BINARY 4

[spiritovod]

I know how it works

Sorry, but considering that you didn't even try to understand my previous post, I hardly doubt.

Let me put it in more simple way: "getbinary MY_BINARY 4" = "getdstring MY_BINARY 4" with current implementation of "binary" term in quickbms. It's just "binary" is being resolved as null-terminated string, so every leading \x0 (it's not zero byte) is considered as delimiter. Otherwise first example from my previous post would work in all possible cases. And findloc is working with strings in that case (it's ever written in the documentation that quickbms handles both binary and strings in the same way), not with byte array. For some functions there is already additional option to ignore delimiter, so you can use null bytes with them as well (for example, putct).

I wonder though, why nobody didn't notice that till now. Null delimiters and leading zeroes are common questions when you're starting to work with binary files (especially binary <-> hex strings representations) in almost all common languages.

-----------------------------------------------

I just came up with one funny example. If you can understand how it works, you can understand the whole representation / conversion thing.

Code: Select all

math TMP = 4
set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
getdstring SEARCH TMP MEMORY_FILE
set SEARCH1 string SEARCH
set SEARCH string ""
math TMP * 2
for i = 0 < TMP
   getvarchr TMP2 SEARCH1 i
   string SEARCH + TMP2
next i
print "Search: %SEARCH%"
string SEARCH h SEARCH
set MEMORY_FILE5 binary "\x01\x01\x02\x02\x00\x01\x00\x02"
findloc RESULT binary SEARCH MEMORY_FILE5 ""
print "Result: %RESULT%"

[Shokoniraya]

i understand, but can you tell me about math TMP * 2 at line 6? was it necessary? because TMP is 4 and with * 2, it reading 8 times and other 4 times is just zero

but after take a look at line 9, you did it in this way: string SEARCH + TMP2
actually, you just attaching decimal numbers together, (you can set all \x00 to \xFF in line 2 and add a print "%TMP2%" after line 8 to see it) like below example

Code: Select all

math TMP = 4
set MEMORY_FILE binary "\xFF\x01\x00\x02\x00\x03"
getdstring SEARCH TMP MEMORY_FILE
set SEARCH1 string SEARCH
set SEARCH string ""
math TMP * 2
for i = 0 < TMP
   getvarchr TMP2 SEARCH1 i
   print "%TMP2%"
   string SEARCH + TMP2
next i

and about line 12: string SEARCH h SEARCH
you want to alterant 01020000 to hex, but we want to find \x00\x01\x00\x02 in MEMORY_FILE5, right? so it ended to \x01\x02?! i think reason is getvarchr, because it just reading decimal numbers and string SEARCH h SEARCH reading string two character by two character and set them as hex, so for example, \xFF\x01\x00\x02 is 4 decimal number and getvarchar and string will make 255102

here a clear examlpe that tells its not working with \x00

Code: Select all

set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
get MEM_SIZE asize MEMORY_FILE

string HEX_STRING = ""
for i = 0 < MEM_SIZE
get BYTE_NUM byte MEMORY_FILE #same job as getvarchr to get every VAR in for-next
string HEX_BYTE p "%02x" BYTE_NUM #so it will make a real hex-string
string HEX_STRING + HEX_BYTE
next i
print "hex string: %HEX_STRING%"
string HEX_STRING h HEX_STRING #and it will make it byte, but its string! so \x00 will be a problem!

set MEMORY_FILE5 binary "\x00\x01\x00\x02\x00\x03"
findloc RESULT binary HEX_STRING MEMORY_FILE5 ""
print "Result: %RESULT%"

null terminate: means reading string until \x00
but getbinary is needed

[Shokoniraya]

sir aluigi, after working on some files with lzo1x, im sure QuickBMS lzo is a buggy version (maybe old and corrupted) and some files will be corrupt in decompressing, some equal bytes in decompressed files always messed up or changed with some uncorrect bytes