QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[aluigi]

Good catch, it seems just a bug in the new "tcc" feature (the runtime C compiler) of quickbms.
The game has tons of files and there is a certain amount of memory consumed and not freed at each cycle so you get the memory allocation error when the process has already allocated some Gb of memory.
I was ready to suggest to use -9 (that limits these types of memory leaks) but quickbms even crashes with it so it looks like a lame bug that I have to solve for the next version.

Unfortunately there are no temporary work-arounds to complete the extraction.

[GenEric]

No sweat, I don't mind waiting for next version. Thanks for your time on this issue. It's been a huge help!

[Acewell]

i think i found a bug with the findloc command, my last entry is ignored
i will attach a sample and the script
*solved and removed*
i didn't see anything in quickbms.txt about there being a limit to how many
times you can use it or where you can/can't use it.

[aluigi]

It's a conflict between the searched string and the name of the first variable.
Fix: replace the name of the first variable (RYHP) with NEXT_RYHP

Yeah I know it's boring but one of the rules of quickbms is that name of variables and constant strings (even the binary ones) are handled in the same way.
Even I forgot that and it required some tests before fixing it

[Acewell]

okay so in newby terms, avoid making the variable and search strings the same, even if one is binary and the other is a string?
thanks

[aluigi]

Exactly, variable names and constant strings (like string or "string" or "\x73tring") MUST be ever different.

[bugmenot]

I think I found a bug with the -P command or the Rpa Renpy Nopython script:

When extracting an RPA file that contains Cyrillic characters in some of the file names they did not extract correctly so tried adding -P UTF8 (unless I'm wrong RPA files are always UTF-8 encoded) but the result was still wrong, next I tried every other UTF and Cyrillic option from this table using both Identifier and .NET Name values, I also tried adding the codepage "utf8" and codepage 1251 string in the BMS script with the same result.

For example the correct file name is: Armature-трет руки_0 as can be seen in the attached file table, but the files are extracted as: Armature-трет СЂСѓРєРё_0, Armature-трет руки_0 or Armature-трет руки_0.

[aluigi]

Yeah that's something that require to be checked for the next release because it appears to be a bug (utf8 is correct but some chars are clearly wrong).
Added in my TODO.

[benfoldsone]

Tried running on OS X El Capitan 10.11.6, and all I get is this text in the Terminal:

Code: Select all

/Users/[redacted]/Downloads/quickbms_0.8.0_macosx/quickbms ; exit;
[computername]~ [redacted]$ /Users/[redacted]/Downloads/quickbms_0.8.0_macosx/quickbms ; exit;
dyld: Symbol not found: __ZdlPvm
  Referenced from: /Users/[redacted]/Downloads/quickbms_0.8.0_macosx/quickbms (which was built for Mac OS X 10.11)
  Expected in: /usr/lib/libstdc++.6.dylib
 in /Users/[redacted]/Downloads/quickbms_0.8.0_macosx/quickbms
Trace/BPT trap: 5
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[aluigi]

Another user reported an unresolved symbol in the exe I uploaded (if that's the exe you are using), but I can't find the post so probably it was by email.
I guess you have to compile it by yourself

[chrrox]

What am i doing wrong to get the encryption key correctly.
If i print the key it is changing but all the log commands I do get the same output.

Code: Select all

putarray 0 0  "\x98\x3b\xa6\xcc\xa7\x52\x32\xf9\x23\xbb\xe8\x7d\x39\x6c\xb7\x4e"
putarray 0 1  "\x13\xc7\xe8\x81\xd9\x8c\x75\x16\xf3\x2f\xbb\xf4\x21\x56\xb1\xa6"
putarray 0 2  "\x38\x50\x7b\x33\xee\xe5\xf0\x53\x4c\x5d\x2f\xc7\xf1\x65\xb8\x4a"
putarray 0 3  "\xc4\x55\x7f\x7c\xba\xb4\x42\x91\xee\x51\x2e\x37\xeb\x23\xaf\x54"
putarray 0 4  "\x23\xeb\x2d\xe6\xf7\x49\x92\x7c\x82\xfd\xec\x8e\xdc\x9f\x3e\xbc"
putarray 0 5  "\xcb\x49\xd8\x7c\xcc\x59\x7b\xca\x2d\xe2\x24\xb7\x19\x36\x37\x4d"
putarray 0 6  "\x2d\x3a\xfb\x28\x54\x21\x50\xb4\xa0\x77\xb8\xbb\xbf\xe8\xb1\x67"
putarray 0 7  "\xe3\x6f\xb2\xef\x65\x2b\xa2\x3a\x58\x2a\xba\x1f\xae\x68\xec\xbc"
putarray 0 8  "\xe7\x1e\x3d\xd6\xf5\xe2\x87\x9f\x68\xa1\x8b\xbb\xc7\xd4\xf2\x7d"
putarray 0 9  "\x3e\x3a\x10\xe4\xc1\x7b\xdf\x72\x39\x46\x40\x16\xfe\x94\x6b\xb5"
putarray 0 10 "\x20\x59\x25\xb9\xa6\x6b\x77\xc0\xbd\x4e\xe0\xbd\x1a\x25\x64\x3b"
putarray 0 11 "\x3f\xd0\x85\x35\x18\xd3\x8c\x59\x89\xc6\xd6\x61\x82\xb8\x5f\x75"
putarray 0 12 "\x25\x2b\xbc\xe5\x3a\xb3\x8b\x75\xb1\x2c\x7a\xdf\x98\xe6\x57\x8b"
putarray 0 13 "\x81\x6a\xcb\xb0\x8b\x17\x15\x10\x40\x53\x7b\x4a\x8a\xce\x77\x18"
putarray 0 14 "\x44\xc9\xcb\xd2\xb1\x76\xed\x2b\x18\xc6\x95\xc9\xd9\xda\x5a\xec"
putarray 0 15 "\x6b\x88\xa6\xd4\x16\x57\x3d\xd0\xb2\x4d\x1f\xf2\x48\x73\x17\x44"

get FILES long MEMORY_FILE
for i = 0 < FILES
    get NAMESZ long MEMORY_FILE
    getdstring NAME NAMESZ MEMORY_FILE
    get SIZE threebyte MEMORY_FILE
    get FLAG byte MEMORY_FILE
    get DUMMY long MEMORY_FILE  # 0xffffffff or zero
    get OFFSET longlong MEMORY_FILE
    math OFFSET += BASE_OFF
    #print "%NAME% %FLAG%"
    set EXT extension NAME
    if EXT == "png"
    print "%NAME% %FLAG%"
    callfunction LAME_GUESS_IDX2 1
    endif
    encryption "" ""
next i

startfunction LAME_GUESS_IDX2
    for KEY_IDX = 0 < 16
        getarray KEY 0 KEY_IDX
        string KEY x KEY
        encryption blowfish KEY "" 0 0x10
        set BNAME NAME
        string BNAME + KEY_IDX
        log BNAME OFFSET 4
    next KEY_IDX
    break
endfunction

[AnonBaiter]

Okay, now this is where I can't grasp your programming script.

So, I'm writing a script to split multiple files out of a .sgb/.sgh file into individual .sgb/.sgh pairs. However, because setting up the header was a bit complex than I thought, I thought of doing this:

Code: Select all

   savepos NAME_OFFPOS1 MEMORY_FILE5
   xmath NAME_OFFPOS_F1 "NAME_OFFPOS1 + 0x10"
   goto NAME_OFFPOS_F1 MEMORY_FILE5
   savepos NAME_OFFPOS_OFF1 MEMORY_FILE5
   putct NAME_NAME1 string -1 MEMORY_FILE5
   put 0 byte MEMORY_FILE5
   savepos NAME_OFFPOS_OFF2 MEMORY_FILE5
   putct NAME_NAME1 string -1 MEMORY_FILE5
   goto NAME_OFFPOS1 MEMORY_FILE5
   put 0 short MEMORY_FILE5
   put 0x3000 short MEMORY_FILE5
   put NAME_OFFPOS_OFF1 long MEMORY_FILE5
   put 0 short MEMORY_FILE5
   put 0 short MEMORY_FILE5
   put NAME_OFFPOS_OFF2 long MEMORY_FILE5

However, this is what actually happens:

Code: Select all

. 00000000000000b8 putct   NAME_NAME1 "MS_01_2" -1
. 00000000000000bf put     0          0x0000000000000000 1
. 00000000000000c0 putct   NAME_NAME1 "MS_01_2" -1
. 00000000000000b8 put     0          0x0000000000000000 2
. 00000000000000ba put     0x3000     0x0000000000003000 2
. 00000000000000bc put     NAME_OFFPOS_OFF1 0x00000000000000b8 4
. 00000000000000c0 put     0          0x0000000000000000 2
. 00000000000000c2 put     0          0x0000000000000000 2
. 00000000000000c4 put     NAME_OFFPOS_OFF2 0x00000000000000c0 4

So you see, the script writes these names through a MEMORY_FILE5 file, but not through the NAME_OFFPOS_F1 offset as expected. Instead, later variables are then written in place of two "putct" lines and one 8-bit(one byte) "put" value on the same MEMORY_FILE5 file.

To elaborate, I want to write these names on an offset that goes after the offset in which a blank space of 16 bytes are reverse for these six "put"(two 16-bit values and one 32-bit value) variables - all into MEMORY_FILE5.

[AnonBaiter]

Turns out it's just me. I started working around this by removing these lines:

Code: Select all

   xmath NAME_OFFPOS_F1 "NAME_OFFPOS1 + 0x10"
   goto NAME_OFFPOS_F1 MEMORY_FILE2

and replacing it with these lines:

Code: Select all

   put 0 long MEMORY_FILE2
   put 0 long MEMORY_FILE2
   put 0 long MEMORY_FILE2
   put 0 long MEMORY_FILE2

Still I won't edit the above post into nothing though, my stupidity will remain there for "historical reasons".

---

Okay, here I am again with my problems with quickBMS.

So I'm pretty close of completing my ".sgh/.sgb/.sgd splitter" script, but the thing is, the archive I want to split is about 512+ files. With MEMORY_FILE stuff(although not with the "get"-like commands thankfully), the script can only generate 256 of these .sgh/.sgb files and it cannot go beyond that number. Here is the proof:

Code: Select all

Error: incomplete input file 1: G:\SONY\PS3\GENJI2\PS3_GAME\USRDIR\DT64\SOUND\G2
DEMO.SGB
       Can't read 111568896 bytes from offset 0000000025602ce0.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 1    42%   627059936  1477753216 . offset 0000000025602ce0

Last script line before the error or that produced the error:
  289 log MEMORY_FILE WAVE_OFFSET1 WAVE_SIZE2 1

So, despite the "unreachable offset" this error tells me I can "reach" that offset just fine through an hex editor, which means that I'm left with a dead-end over an archive whose data is entirely intact.

So to summarize, only 512 files(514 in G2DEMO.SGB/G2DEMO.SGH this case) are allowed to be written while on MEMORY_FILE mode and any number beyond that is overkill for quickBMS at this point. Is there any way around this?

[spider91]

Hi, open command in new version of quickbms doesn't work well

Code: Select all

Open FOLDER NAME [FILENUM] [EXISTS]

When i use code like this

Code: Select all

open "." NAME 2 CHECK
 if CHECK == 0
    some actions
else
    some actions
endif

i get messages that ask to press Y/N for each file (https://puu.sh/wHJv5/2b271a821e.png). Older versions of quickbms didn't have this "bug" and never asked any confirmation. Please fix it, cause it makes [EXISTS] argument useless.

[spider91]

Findloc has a searching bug for some offsets - https://puu.sh/wJwIr/bfc1ff14e3.png

I've tried to search that string apart with script

Code: Select all

string ID p= "%08x" 0xd0006842
string ID += "="
#goto 1
FindLoc STR_OFF string "d0006842="

and this is what i've got - https://puu.sh/wJwYg/3952738b94.png
As you can see when i search from beginning of file bms can't find anything, but if i change position to 1 (goto 1) it works fine.

P.S.
0.7.7 and 0.8.0 gives the same result.

[aluigi]

The EXISTS field of the Open command gave me no problem till now so I suspect there is something else, maybe are you using the reimport feature or the -w write mode?
The "do you want to create it from scratch" is used just when you must write files and not during the normal reading operations:

Code: Select all

    if(g_write_mode) {
        filez->fd = xfopen(fname, "r+b");    // do NOT modify, it must be both read/write
        if(!filez->fd) {
            if(g_reimport) {
                if(error) STD_ERR(QUICKBMS_ERROR_FILE_WRITE);
                return -1;
            } else {
                fprintf(stderr, "\n"
                    "- the file %s doesn't exist.\n"
                    "  Do you want to create it from scratch (y/N)?\n"

Regarding the findloc problem you are right, if the string starts at offset 0x9fdc it's not seen, while it works if it's in other positions... good catch!

[AnonBaiter]

The "do you want to create it from scratch" is used just when you must write files and not during the normal reading operations:

Code: Select all

    if(g_write_mode) {
        filez->fd = xfopen(fname, "r+b");    // do NOT modify, it must be both read/write
        if(!filez->fd) {
            if(g_reimport) {
                if(error) STD_ERR(QUICKBMS_ERROR_FILE_WRITE);
                return -1;
            } else {
                fprintf(stderr, "\n"
                    "- the file %s doesn't exist.\n"
                    "  Do you want to create it from scratch (y/N)?\n"

I've never seen that before. Is this the solution to my problem or are you going with this "feature" on the next version?
(Pardon my ignorance.)

[aluigi]

That's for spider91.

I gave a quick look at your post but... headache

[spider91]

The EXISTS field of the Open command gave me no problem till now so I suspect there is something else, maybe are you using the reimport feature or the -w write mode?
The "do you want to create it from scratch" is used just when you must write files and not during the normal reading operations:

Code: Select all

    if(g_write_mode) {
        filez->fd = xfopen(fname, "r+b");    // do NOT modify, it must be both read/write
        if(!filez->fd) {
            if(g_reimport) {
                if(error) STD_ERR(QUICKBMS_ERROR_FILE_WRITE);
                return -1;
            } else {
                fprintf(stderr, "\n"
                    "- the file %s doesn't exist.\n"
                    "  Do you want to create it from scratch (y/N)?\n"

Regarding the findloc problem you are right, if the string starts at offset 0x9fdc it's not seen, while it works if it's in other positions... good catch!

Not reimport, but yes, i use -w mode. Anyway i don't think that's a good solution even with write mode, cause we have [EXISTS] argument specially to check it.
Or maybe it would be better this message to appear only when [EXISTS] argument is not used?

[aluigi]

That code probably doesn't consider the EXISTS field (variable "error" in the fdnum_open function) in a correct way or, more probably, it was meant for a different usage that I don't remember.
Anyway yes I can easily fix it by adding an "if(!error) return -1;" just before fprintf.