QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[aluigi]

Ok so it's definitely that one the problem.
Even if callfunction is invoked without saving variables, it still calls two internal functions recursively (start_bms and CMD_Function_func).

I will think if there is something I can implement to "reduce" this issue.

[aluigi]

The following is an example script for testing the problems of callfunction (without preserving variables which is memory consuming without the -9 option):

Code: Select all

math COUNTER = 0
callfunction FUNCTION 1
startfunction FUNCTION
    print "%COUNTER%"
    math COUNTER + 1
    callfunction FUNCTION 1
endfunction

It can do over 7000 iterations before consuming the stack.

Some years ago I played a lot with the "-Wl,--stack,BYTES" option for the gcc linker and at the end I opted for -Wl,--stack,4194304.
There is a reason behind that number (4Mb) based on all my tests but I don't remember if it was related to a limitation of the compiler or an incompatibility on older Windows platforms.

There is for sure a problem in getarray because it may consume about 600 bytes for nothing (easy to fix), but it would not increase much the number of cycles.

*edit*: I even tried to remove all those "goto" in start_bms() in bms.c but the result is the same.

[aluigi]

Anyway I just updated quickbms_exe.zip.
The cycles are the same as before, you may probably get few more due to the less space used for variable_copy invoked by getarray/putarray... but it would not solve the problem.

I have no other ideas at the moment.

[tbmq008]

alright
what i do need to tell you though is that this part

Code: Select all

   goto 0
   math vol_ver = 0
   math c_01 = 0
   do
      savepos temp_02
      get vol_check_01 long
      get vol_check_02 long
      if vol_check_01 == 0xacb990ad
         math c_01 + 1
         putarray c_01 0 temp_02
         if vol_check_02 == 0x00020002
            math vol_ver = 1
         endif
         if vol_check_02 == 0x00030001
            math vol_ver = 2
         endif
      endif
      math temp_02 += 0x800
      goto temp_02
   while temp_02 != 0x2000000

is somehow causing a bug in the new beta.
like this.

Code: Select all

  offset   filesize   filename
--------------------------------------
. 00000000 get     is_vol     "vol" -1002
.
*cut by aluigi*
.

-------------------
*EXCEPTION HANDLER*
-------------------
An error or crash occurred:

*EH* ExceptionCode      c00000fd stack overflow
*EH* ExceptionFlags     00000000
*EH* ExceptionAddress   75C97FC7
                        75C30000 + 00067fc7 msvcrt.dll
*EH* NumberParameters   00000002
*EH*                    00000000
*EH*                    03862000

Last script line before the error or that produced the error:
  17  do
  coverage file 0     0%   0          4261412864 . offset 00000000
.

[aluigi]

Yeah sorry and thanks for noticing it.
Removing "goto" from a function ever leads to problems
The exe is now fixed.

[tbmq008]

alright, with that being fixed i can now focus on trying to fix my script.
as i try to brave myself through as many stack overflow errors as possible.

[aluigi]

Two funny things I noticed:

- quickbms_4gb_files uses less stack despite adopting 64bit integers since it can go over 10k cycles instead of 7k of quickbms.exe

- the new beta without "goto" performs worsen than the previous one (still in quickbms_beta.zip) that used the old goto solution in start_bms, the difference is about 200 cycles more in the old one

[aluigi]

Not sure if you read that topic already, but there is possible bug with quickbms_hash or escapes in general: viewtopic.php?f=9&t=14485

The problem was in the script.
This is a simple test to understand the behavior of "binary":

Code: Select all

set VAR string "asdf\x22asdf"
print "%VAR%"
set KEY binary VAR
print "%KEY%"

I know, handling non-string variables and content is ever a headache with quickbms due to the zeroes (even if this is not the case) but the beta solves these problems with String:

Code: Select all

set VAR binary "asdf\x00asdf"
print "%VAR%"
String KEY  = VAR
print "%KEY|dump%"
String KEY 0= VAR
print "%KEY|dump%"

[spiritovod]

So it's old good zeroes. While the base artemis_engine script now works fine with the game, I expect it will fail if some \x00 will slip in (though it should not, since it's a hash). Sorry, but did you consider to add some basic explanation about how zeroes works in quickbms to the documentation? Because such examples, when a script works fine with 2 out of 3 files with exactly the same structure, are... you know... confusing. For example, in UE4 there are guids instead of hashes, which can easily contains "\x00\x00" - and thus you can't use VAR h/b VAR in expected way for hex representation and you should use something like this instead (or any other workaround). It's like all related examples in the documentation doesn't consider that 00 will exist in input/output variables.

[aluigi]

The only problem with zeroes is related to the visualization of binary data and strings operations, and maybe copying binary variables to others.

In the example of the HASH you mentioned, there is absolutely no problem using "getdstring HASH 16" to read the data and it's stored and handled with the zeroes, so "putdstring HASH 16" perfectly works.
All the variables in quickbms are in binary form and include zeroes.

The problem is how to visualize that data to something readable for the user.
This one works: print "%HASH|dump%" and print "%HASH|hex 16%"
Also this one works: slog "file.txt" -1 16 (the output is with \ escapes)
This one does NOT work or not work well: string OUTPUT b HASH or string OUTPUT B HASH

In quickbms 0.11 this one works: string OUTPUT 0b HASH ; print "%OUTPUT%"

So, yeah, there is not really any problem.
All the constant strings in the bms scripts are handled as strings without any escape, those with the \ escape are tagged as "cstring". It's something inherithed from the original BMS language.
There is an option for both command-line and quickbmsver that forces all the constant strings to be cstring like any other programming language.

[spiritovod]

Yes, and my point was to put those kind of examples somewhere in the documentation around the header. Because null-terminated strings only exist in С, so if you're switching from java/python to quickbms, those things are not obvious. Btw, I don't quite understand why you've mentioned cstring - if you mean -J option, does it works like usual string, but with escapes? I mean, how escapes works has nothing to do with such kind of examples in other languages, because the problem is caused by null-termination of strings which doesn't exist in vanilla version of most other programming languages.

Again, I'm sorry, but don't you think that you shouldn't explain how quickbms works internally to end users?
Let's take a random user - he want to convert a binary string to hex string with quickbms. He go to documentation and see that strings and binary are handled almost equally. And there is option to convert string to hex via b/B option. But like you've said by yourself, it will not work well in certain cases (and there are a lot of them if you're dealing with binary sequences). Do you mention that there are exceptions? No. Is that somehow explained in documentation? No. Should that user start to learn С to understand things? Um... probably not.

[aluigi]

The original BMS language was very limited.
There was nothing like backslash escaped strings or similar, so what I did was simply improving the language by keeping most of its syntax.
And this process took many years since it was based on my personal needs with new formats.

That's why "string VAR + VAR" where VAR has 0x00 bytes in it, has a different result than what you get by doing the same on languages like java.

Each version I'm trying to improve these mechanisms and, indeed, now (beta) "string VAR 0+ VAR" works with binary data too.

The documentation already contains all the references to these limitations.
I improved it in the beta so maybe it's a bit more clear there.

quickbms started as a simple open source tool for the BMS language for my personal usage to avoid to make a standalone tool for each format, I didn't expect it to become now so diffused among users and modders for writing scripts.
If I knew that in advance maybe I would have taken difference choices

Ah, yes, with -J any string you use in the script can be escaped:
set VAR string "hello\x20\x22world\x22"

[Shokoniraya]

dear spiritovod
both string and binary works fine, it depends to user
and if a user wants a dufferent result, then it's better to do it by hand, QuickBMS already working as a binary tool as well to do things by hand, even converting unicode strings by hand

[spiritovod]

aluigi, you're right, next 0.11.x version should be a big improvement in handling binary sequences in more "streaming" way, like in common languages. And it's not like I'm complaining, quickbms is a great tool to deal with some daily tasks and especially all kind of encryption algorithms
I've checked papers for latest beta and couldn't find anything related to zeroes usage and limitations in strings. Maybe it's worth to add something like "if you expect null bytes in the string, it's better to use binary mode with 0 prefix" to related functions of String command or String description. And maybe it's just me, but old descriptions and examples for base string functions (+ - ^ $ etc) were much more clearer than current ones.

Shokoniraya, that's not actually true. "By hand" doesn't mean you should guess how exactly a function works, compared to its description, and the only one who can explain such misunderstandings would be aluigi. But a simple example in the documentation can resolve a lot of those questions.

[Shokoniraya]

sir aluigi, there is a problem with creating a MEMORY_FILE by string

Code: Select all

get FILE_NAMER filename
set MEMORY_FILE string FILE_NAMER
get MEM_SIZE asize MEMORY_FILE
log file_name 0 MEM_SIZE MEMORY_FILE

output file has many zero bytes after main string

[aluigi]

@Shokoniraya
That's one of the weird things that rarely happen in quickbms due to some internal work-arounds.
It's a situation that should never happen since put/putdstring/putct are the best way for handling the writes to MEMORY_FILE (which is a file indeed).

Many years ago I decided to add a security layer to quickbms due to the various external libraries and code I used (and for my buggy code too obviously).
The source code is stored in extra\xalloc.c and it's a complete and transparent replacement of malloc/calloc/realloc/free.
In short every time some memory is allocated, quickbms puts it between two read-only pages that will trigger an exception if you try to read/write before/after your memory.
The first page also contains important information about the allocated memory.

Every page is 4kb, so if you allocate 1 byte quickbms will take 12kb (4+4+4).
It's a good thing for security and usually it doesn't hit much on the performances, obviously it takes more memory but it's still "ok" during normal usage.
If you use many recursive functions it's going to take tons of memory.

This feature was part of an experiment that I decided to implement in quickbms since it was perfect for it.

Long story short, the variables shorter than 271 characters are directly stored in the global array of variables without doing any allocation.
Longer variables require memory allocation.

That's the reason why you get that result but it's also a bug that I'm going to check since MEMORY_FILE should get the real length and not that "internal" length.

Ah, that allocation protection can be disabled with the -9 option.
It's usually suggested in some specific cases and requires user's acknowledgement if set via quickbmsver.

[aluigi]

And maybe it's just me, but old descriptions and examples for base string functions (+ - ^ $ etc) were much more clearer than current ones.

I will check what I can do for the description of the String examples, I hope the following script can be helpful for testings all the various outputs:

Code: Select all

# String command test

set VAR1 string "MyStringExampleString!"
set VAR2 string "STRING"
math NUM = 3
math NEG = -3

strlen TMP VAR1
print "VAR1 is %VAR1%"
print "VAR1 is %TMP% bytes"
print "VAR2 is %VAR2%"
print "NUM  is %NUM%"
print "NEG  is %NEG%"
print "."

print "+   append"
set VAR string VAR1
string VAR + VAR2
print "      %VAR%"
set VAR string VAR1
string VAR + NUM
print "      %VAR%"
set VAR string VAR1
string VAR + NEG
print "      %VAR%"

print "-   remove, truncate"
set VAR string VAR1
string VAR - VAR2
print "      %VAR%"
set VAR string VAR1
string VAR - NUM
print "      %VAR%"
set VAR string VAR1
string VAR - NEG
print "      %VAR%"

print "^   xor"
set VAR string VAR1
string VAR ^ VAR2
print "      %VAR%"
set VAR string VAR1
string VAR ^ NUM
print "      %VAR%"
set VAR string VAR1
string VAR ^ NEG
print "      %VAR%"

print "<   strrstr + var2 (before)"
set VAR string VAR1
string VAR < VAR2
print "      %VAR%"
set VAR string VAR1
string VAR < NUM
print "      %VAR%"
set VAR string VAR1
string VAR < NEG
print "      %VAR%"

print "*   replicate"
set VAR string VAR1
string VAR * VAR2
print "      %VAR%"
set VAR string VAR1
string VAR * NUM
print "      %VAR%"
set VAR string VAR1
string VAR * NEG
print "      %VAR%"

print "%%   strstr (before), truncate, mod"
set VAR string VAR1
string VAR % VAR2
print "      %VAR%"
set VAR string VAR1
string VAR % NUM
print "      %VAR%"
set VAR string VAR1
string VAR % NEG
print "      %VAR%"

print "&   strstr"
set VAR string VAR1
string VAR & VAR2
print "      %VAR%"
set VAR string VAR1
string VAR & NUM
print "      %VAR%"
set VAR string VAR1
string VAR & NEG
print "      %VAR%"

print "|   strstr + var2"
set VAR string VAR1
string VAR | VAR2
print "      %VAR%"
set VAR string VAR1
string VAR | NUM
print "      %VAR%"
set VAR string VAR1
string VAR | NEG
print "      %VAR%"

print "$   strrstr"
set VAR string VAR1
string VAR $ VAR2
print "      %VAR%"
set VAR string VAR1
string VAR $ NUM
print "      %VAR%"
set VAR string VAR1
string VAR $ NEG
print "      %VAR%"

print "!   strrstr + var2"
set VAR string VAR1
string VAR ! VAR2
print "      %VAR%"
set VAR string VAR1
string VAR ! NUM
print "      %VAR%"
set VAR string VAR1
string VAR ! NEG
print "      %VAR%"

print ">   strrstr (before)"
set VAR string VAR1
string VAR > VAR2
print "      %VAR%"
set VAR string VAR1
string VAR > NUM
print "      %VAR%"
set VAR string VAR1
string VAR > NEG
print "      %VAR%"

[aluigi]

I just post it here, it's not an error but rather a "feature".
Long story short some instructions like calldll load the MEMORY_FILE from the current position rather than from its beginning.

The reason is that in some situations it's very useful to seek to a specific offset of the memory file and using it from that position.

I now realized that also the NameCRC instruction and probably others do the same.

For example I built a MEMORY_FILE with the list of hashes and filenames, and it didn't work.
The reason is that I had to use "goto 0 MEMORY_FILE" first.

Keep it in mind!

[gameside]

Hi, its possible to remove some byte from a file?
I mean like put or putdstring, but instead of replacing bytes, remove them

[aluigi]

As far as I remember there is no easy way to do that.
Something like that is automatically done when using reimport3 and your reimported file is smaller.
Your best alternative is just dumping a new file with the desired size.