QuickBMS errors [programming, scripting, quickbms.exe tool... NOT games]

[AnonBaiter]

So, is it normal to have an memory allocation problem error? Is this error related to my memory space, or my hard disk space?

[aluigi]

Just a generic quick description of the two main problems.
tl;dr it's a problem in the scripts

"impossible to write 0xf... bytes" are problems in the scripts or formats that are not 100% covered by the script, basically it's a calculation error where the next offset is before the current one and so it results in a huge number (negative), that may happen in offset-only based scripts where the size is calculated by the substraction of the current offset from the next one.

"memory allocation problem" is more generic and may be similar to the previous one, usually it's a problem caused by a signed comparison (>=) instead of the unsigned one (u>=) or any other problem where the script wants to dump a big file or use lot of memory.

In both the cases the script doesn't match the format of the input file.
When you encounter these errors you should post in the thread of the game so that we can update the script.

[AnonBaiter]

Well, I think I understand now... Thanks.

[AnonBaiter]

Right off the bat I got this:

Code: Select all

. 0000000000009282
. 0000000000024a50 get     SIZE       0x00000000000d1ec4 4
. 0000000000024a54 get     OFFSET     0x0000000000006613 4
. 0000000000024a58 get     LANG_ID    0x00000000ffffffff 4
. 0000000000024a5c get     ZSIZE      0x0000000000000000 4
- enter in folder E:\DeusEx\DeusEx\data\content

- error in src\cmd.c line 3082: CMD_Open_func()
Error: No such file or directory

Last script line before the error or that produced the error:
  54  open FDDE TMP 1

And that's from the new version, no less!
By the way, this is the script I'm using for the file:
http://aluigi.altervista.org/bms/deus_ex_human_revolution.bms

[aluigi]

Interesting, I'm checking it just now.
It's a problem that currently happens only with this script because I already made some simple tests and it was all correct.
It's probably related to the new feature I added for working with non-english paths and filenames.

[aluigi]

Problem found.
It's related to the endianess.

[AnonBaiter]

So... what should I do about this?

[aluigi]

Wait 15 minutes.

[aluigi]

Fixed in 0.7.6a, thanks for the report.

[TheUkrainianBard]

I finally tried "comtype ps_lz77" but it threw an error on one of the files I've attached in this post.
I've cut everything before the "LZ77" ID in hex editor. I've used this script.

[aluigi]

I guess the problem is the endianess because face.lz7 is big endian and works while Obj011.tex is little endian and I get a "guard page violation" with quickbms 0.7.6a (which means that it's reading the data before/after the buffer).
The reason why the algorithm doesn't work even when you skip the header it's because it uses two inputs and the only way to know where the second input starts is reading the header.

[TheUkrainianBard]

Where can I find ps_lz77 algo code in sources?
Is it possible to make the algorithm use the script endianness?

Or, as a suggestion, maybe don't read the header inside the algo and use set (manually or through scripts) SIZE, ZSIZE and some OFFSET (to get past the flags to literals/codepairs - it can be calculated though), as these values are right after the "LZ77" ID in files. Akin to zlib.

Maybe an example will help:

Code: Select all

quickbmsver "0.7.5"
endian little
idstring "LZ77"
comtype ps_lz77
get SIZE long # get uncompressed size at 0x4
get ZSIZE long # get flag count at 0x8
get OFFSET2 long # get the relative (vs "LZ77" ID) start point of literals/codepairs at 0xC
savepos OFFSET1 # save the start point of flags (should be at +0x10 vs "LZ77" ID)
get NAME basename
get EXT extension
string NAME + "_unpack."
string NAME + EXT
Clog NAME OFFSET ZSIZE SIZE

[aluigi]

The endianess based on the one of the script is the best solution.
Honestly I don't know why I opted for the hardcoded big-endian, the post you made was full of files and scripts so it made just chaos in my mind... probably I just didn't see the attachment with the other Obj*.tex files.

Anyway quickbms 0.7.5 has been released over one month ago, had you no occasion to check this before?
I ask because I have released quickbms 0.7.6a just 2 days ago and this is the second report about a problem with a native compression algorithm after the release.

[TheUkrainianBard]

I did check, but I did not report. Like "eh, it's not working, I'll stick/fall back to the slowest implementation ever" (as I did here) and forgot about it.
Sorry.

[aluigi]

That's bad, there is still no plan about when quickbms 0.7.7 will be released.
Currently there are only two things in my todo, one is this ps_lz77 and the other one is for lbalzss* that has been already fixed inside the scripts so zero priority (through "comtype calldll" and the fixed function in the script).

There are work-arounds also for this ps_lz77, for example a script to automatically convert the endianess in 4 lines plus the -E option of quickbms, or a script that converts the endianess and puts it in a memory file and so on.
In short:

Code: Select all

findloc OFFSET binary "LZ77"
get SIZE asize
math SIZE - OFFSET

# work-around for quickbms <= 0.7.6a
log MEMORY_FILE OFFSET SIZE
idstring MEMORY_FILE "LZ77"
get TMP1 long MEMORY_FILE
get TMP2 long MEMORY_FILE
get TMP3 long MEMORY_FILE
reverselong TMP1
reverselong TMP2
reverselong TMP3
putvarchr MEMORY_FILE 4 TMP1 long
putvarchr MEMORY_FILE 8 TMP2 long
putvarchr MEMORY_FILE 12 TMP3 long
comtype ps_lz77

get NAME basename
get EXT extension
string NAME + "_unpack."
string NAME + EXT
clog NAME 0 SIZE SIZE MEMORY_FILE

Please note that in any case you were using the script in a wrong way because you have offset 0 specified in your error while the correct offset was 0xc!

[AnonBaiter]

Hi.

I want to report that your comtype_scan2.bat file won't work with output files that are over 2gb.

Code: Select all

QuickBMS generic files extractor and reimporter 0.7.6a
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Aug 22 2016 - 18:08:37)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- open input file E:\MAME ROMs\tvcapcom\tvc_read1.u14
- open script E:\quickbms_scripts\comtype_scan2.bms
- set output folder E:\MAME ROMs\tvcapcom\tvc_read

  offset   filesize   filename
--------------------------------------
- SCRIPT's MESSAGE:
  test algorithm number 1: ZSIZE 553650328, SIZE -1811895328

  00000000 2483071968 1.dmp

Error: impossible to write 0x9400abe0 bytes (total 0x9400abe0)
       Check your disk space

Last script line before the error or that produced the error:
  27  clog NAME 0 ZSIZE SIZE

QuickBMS generic files extractor and reimporter 0.7.6a
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Aug 22 2016 - 18:08:37)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- open input file E:\MAME ROMs\tvcapcom\tvc_read1.u14
- open script E:\quickbms_scripts\comtype_scan2.bms
- set output folder E:\MAME ROMs\tvcapcom\tvc_read

  offset   filesize   filename
--------------------------------------
- SCRIPT's MESSAGE:
  test algorithm number 2: ZSIZE 553650328, SIZE -1811895328

  00000000 2483071968 2.dmp

Error: impossible to write 0x9400abe0 bytes (total 0x9400abe0)
       Check your disk space

Last script line before the error or that produced the error:
  27  clog NAME 0 ZSIZE SIZE

QuickBMS generic files extractor and reimporter 0.7.6a
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Aug 22 2016 - 18:08:37)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- open input file E:\MAME ROMs\tvcapcom\tvc_read1.u14
- open script E:\quickbms_scripts\comtype_scan2.bms
- set output folder E:\MAME ROMs\tvcapcom\tvc_read

  offset   filesize   filename
--------------------------------------
- SCRIPT's MESSAGE:
  test algorithm number 3: ZSIZE 553650328, SIZE -1811895328

  00000000 2483071968 3.dmp

Error: impossible to write 0x9400abe0 bytes (total 0x9400abe0)
       Check your disk space

Last script line before the error or that produced the error:
  27  clog NAME 0 ZSIZE SIZE

QuickBMS generic files extractor and reimporter 0.7.6a
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Aug 22 2016 - 18:08:37)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- open input file E:\MAME ROMs\tvcapcom\tvc_read1.u14
- open script E:\quickbms_scripts\comtype_scan2.bms
- set output folder E:\MAME ROMs\tvcapcom\tvc_read

  offset   filesize   filename
--------------------------------------
- SCRIPT's MESSAGE:
  test algorithm number 4: ZSIZE 553650328, SIZE -1811895328

  00000000 2483071968 4.dmp

Error: impossible to write 0x9400abe0 bytes (total 0x9400abe0)
       Check your disk space

Last script line before the error or that produced the error:
  27  clog NAME 0 ZSIZE SIZE

[aluigi]

If you use the scanner with files of over few megabytes, it means you have no idea of what you are doing.
I'm serious, do NOT use the comtype scanner if you don't know what it's meant for because it's an advanced feature.

[AnonBaiter]

I think I might have some problems with extracting files that use unusual characters(such as japanese):


Perhaps it's just my computer or something... any ideas?

[aluigi]

viewtopic.php?p=17542#p17542

[zaphiel]

I'm getting an error with trying to extract unity 5 (I think) resS files. I'm getting the memory allocation problem with 0.7.6a and using the 0.2.4 unity script.

Not sure if you need the whole extract log, but here it is.

Code: Select all

QuickBMS generic files extractor and reimporter 0.7.6a
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Aug 22 2016 - 18:08:37)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- GUI mode activated, remember that the tool works also from command-line
  where are available various options like folder scanning, filters and so on

- select the BMS script to use
- select the input archives/files to extract, type * or "" for whole folder and subfolders
- select the output folder where extracting the files
- open input file C:\Users\Frances\Downloads\quickbms\New folder\sharedassets1.assets
- open script C:\Users\Frances\Downloads\quickbms\unity.bms
- set output folder C:\Users\Frances\Downloads\quickbms\New folder\New folder

  offset   filesize   filename
--------------------------------------
- SCRIPT's MESSAGE:
  5.4.2f1

  000022c4 1172       sharedassets1~sharedassets1_0.bin

- The following output file already exists:
  sharedassets1~sharedassets1_0.bin
  Do you want to overwrite it?
    y = overwrite (you can use also the 'o' key)
    n = skip (default, just press ENTER)
    a = overwrite all the files without asking
    r = automatically rename the files with the same name
    s = skip all the existent files without asking

s
  00002764 100        sharedassets1~default.tex
  000027d4 100        sharedassets1~leopard.tex
  0000284c 100        sharedassets1~razorback_happy.tex
  000028cc 100        sharedassets1~falkner_annoyed_blush.tex
  00002940 100        sharedassets1~falkner_sad.tex
  000029b4 100        sharedassets1~default.tex
  00002a38 100        sharedassets1~falkner_neutral_closedeyes.tex
  00002aac 100        sharedassets1~tirune.tex
  00002b2c 100        sharedassets1~bittersweet_closedeyes.tex
  00002b9c 100        sharedassets1~smile.tex
  00002c14 100        sharedassets1~falknersize_2.tex
  00002c8c 100        sharedassets1~bg_tirune_dragon.tex
  00002d04 100        sharedassets1~bg_tirune_end.tex
  00002d80 100        sharedassets1~razorback_default.tex
  00002dfc 100        sharedassets1~faraway_blush.tex
  00002e78 100        sharedassets1~falkner_mansplain.tex
  00002ef8 100        sharedassets1~default_closedeyes.tex
  00002f74 100        sharedassets1~kuya_embarrassed.tex
  00002ff4 100        sharedassets1~crow_earnest_closedeyes.tex
  00003068 100        sharedassets1~kuya_cocky.tex
  000030dc 100        sharedassets1~Square.tex
  0000314c 100        sharedassets1~options.tex
  000031bc 100        sharedassets1~shouty.tex
  0000322c 100        sharedassets1~bg_kuya.tex
  000032a0 100        sharedassets1~bg_falkner.tex
  0000331c 100        sharedassets1~karro_neutral.tex
  00003398 100        sharedassets1~Full Dialogue Box.tex
  00003410 100        sharedassets1~cave_warm.tex
  0000348c 100        sharedassets1~littleprince_mad.tex
  00003504 100        sharedassets1~falknersize_4.tex
  00003578 100        sharedassets1~snowstorm.tex
  000035f8 100        sharedassets1~crow_neutral_blush.tex
  0000366c 100        sharedassets1~shoujo.tex
  000036dc 100        sharedassets1~fox_3.tex
  0000375c 100        sharedassets1~crow_genuine_closedeyes.tex
  000037d0 100        sharedassets1~crow_oldman.tex
  0000384c 100        sharedassets1~falkner_annoyed.tex
  000038c0 100        sharedassets1~fastforward.tex
  00003930 100        sharedassets1~cave.tex
  000039ac 100        sharedassets1~flustered_blush.tex
  00003a28 100        sharedassets1~knight_closedeyes.tex
  00003aa0 100        sharedassets1~pilgrim_1.tex
  00003b24 100        sharedassets1~kuya_neutral_closedeyes.tex
  00003ba4 100        sharedassets1~kuya_romance_closedeyes.tex
  00003c1c 100        sharedassets1~falknersize_1.tex
  00003c9c 100        sharedassets1~snowymountain_default.tex
  00003d0c 100        sharedassets1~heart3.tex
  00003d94 100        sharedassets1~falkner_neutral_closedeyes_blush.tex
  00003e0c 100        sharedassets1~dragon_intense.tex
  00003e8c 100        sharedassets1~falkner_neutral_blush.tex
  00003f08 100        sharedassets1~littleprince_haughty.tex
  00003f78 100        sharedassets1~wolf.tex
  00003fec 100        sharedassets1~logroll.tex
  00004060 100        sharedassets1~falkner_end.tex
  000040e4 100        sharedassets1~kuyasprite_rosemarycarry.tex
  00004158 100        sharedassets1~nude_kuya2.tex
  000041d0 100        sharedassets1~nude_kuya1.tex
  00004244 100        sharedassets1~fox_1.tex
  000042b8 100        sharedassets1~tirunefinal.tex
  0000433c 100        sharedassets1~snowymountain_morning.tex
  000043b0 100        sharedassets1~pilgrim_2.tex
  00004430 100        sharedassets1~kuya_sad_closedeyes.tex
  000044b0 100        sharedassets1~bittersweet_blush.tex
  00004524 100        sharedassets1~knight.tex
  000045a4 100        sharedassets1~kuya_cocky_closedeyes.tex
  00004620 100        sharedassets1~razorback_hurt_two.tex
  00004698 100        sharedassets1~flustered.tex
  00004710 100        sharedassets1~NamePlate.tex
  0000478c 100        sharedassets1~falkner_neutral.tex
  0000480c 100        sharedassets1~kuya_embarrassed_blush.tex
  00004880 100        sharedassets1~bg_canyon.tex
  000048f4 100        sharedassets1~river.tex
  00004960 100        sharedassets1~sad.tex
  000049d8 100        sharedassets1~crow_earnest.tex
  00004a5c 100        sharedassets1~rosemarysprite_battle.tex
  00004ae0 100        sharedassets1~crow_bittersweet_closedeyes.tex
  00004b5c 100        sharedassets1~cave_warm_magic.tex
  00004bd8 100        sharedassets1~razorback_hurt_one.tex
  00004c50 100        sharedassets1~bittersweet.tex
  00004cd8 100        sharedassets1~falkner_mansplain_closedeyes.tex
  00004d4c 100        sharedassets1~combined.tex
  00004dc0 100        sharedassets1~largesidetab.tex
  00004e40 100        sharedassets1~shouty_closedeyes.tex
  00004ec0 100        sharedassets1~falkner_sad_blush.tex
  00004f40 100        sharedassets1~littleprince_neutral.tex
  00004fbc 100        sharedassets1~kuya_angry_blush.tex
  0000502c 100        sharedassets1~heart2.tex
  000050a8 100        sharedassets1~dragon_noeyebrows.tex
  00005120 100        sharedassets1~pilgrim_4.tex
  0000519c 100        sharedassets1~river_snowstorm.tex
  00005214 100        sharedassets1~kuya_cocky_blush.tex
  00005294 100        sharedassets1~falkner_sad_closedeyes.tex
  0000530c 100        sharedassets1~kuya_sad_blush.tex
  00005378 100        sharedassets1~crow.tex
  000053ec 100        sharedassets1~angry.tex
  0000545c 100        sharedassets1~crow_end.tex
  000054d4 100        sharedassets1~windingmountain.tex
  00005550 100        sharedassets1~faraway_closedeyes.tex
  000055d4 100        sharedassets1~windingmountain_night.tex
  0000564c 100        sharedassets1~falknersize_3.tex
  000056c8 100        sharedassets1~falkner_insincere.tex
  00005740 100        sharedassets1~sad_blush.tex
  000057b8 100        sharedassets1~smile_blush.tex
  0000582c 100        sharedassets1~upset.tex
  000058a8 100        sharedassets1~crow_genuine_blush.tex
  00005930 100        sharedassets1~kuya_angry_blush_closedeyes.tex
  000059b0 100        sharedassets1~ranasprite_default.tex
  00005a24 100        sharedassets1~bg_crow.tex
  00005a98 100        sharedassets1~crow_neutral.tex
  00005b20 100        sharedassets1~bittersweet_closedeyes_blush.tex
  00005ba4 100        sharedassets1~falkner_insincere_blush.tex
  00005c18 100        sharedassets1~knight_blush.tex
  00005c8c 100        sharedassets1~tirune_2.tex
  00005d00 100        sharedassets1~pilgrim_3.tex
  00005d74 100        sharedassets1~dragon.tex
  00005de8 100        sharedassets1~crow_genuine.tex
  00005e5c 100        sharedassets1~faraway.tex
  00005ed0 100        sharedassets1~crow_laugh.tex
  00005f50 100        sharedassets1~snowymountain_night.tex
  00005fcc 100        sharedassets1~falkner_spear.tex
  00006044 100        sharedassets1~smile_closedeyes.tex
  000060c4 100        sharedassets1~kuya_angry_closedeyes.tex
  00006134 100        sharedassets1~heart1.tex
  000061a8 100        sharedassets1~kuya_neutral.tex
  00006220 100        sharedassets1~shouty_blush.tex
  0000629c 100        sharedassets1~default_blush.tex
  00006314 100        sharedassets1~default_blush.tex
  00006388 100        sharedassets1~kuya_romance.tex
  000063fc 100        sharedassets1~shoujo.tex
  0000647c 100        sharedassets1~falkner_sincere_blush.tex
  000064f4 100        sharedassets1~crow_bittersweet.tex
  0000656c 100        sharedassets1~nameplateblur.tex
  000065dc 100        sharedassets1~falkner.tex
  00006648 100        sharedassets1~kuya.tex
  000066d0 100        sharedassets1~bg_cliffoverlookingstarlight.tex
  00006744 100        sharedassets1~kuya_sad.tex
  000067b0 100        sharedassets1~owl.tex
  00006834 100        sharedassets1~crow_neutral_closedeyes.tex
  000068b4 100        sharedassets1~knight_blush_closedeyes.tex
  00006924 100        sharedassets1~fox_2.tex
  0000699c 100        sharedassets1~sad_closedeyes.tex
  00006a1c 100        sharedassets1~windingmountain_storm.tex
  00006a94 100        sharedassets1~falkner_sincere.tex
  00006b08 100        sharedassets1~angry_blush.tex
  00006b80 100        sharedassets1~kuya_angry.tex
  00006bf4 100        sharedassets1~tirune.tex
  00006c68 100        sharedassets1~nude_kuya3.tex
- enter in folder C:\Users\Frances\Downloads\quickbms\New folder
- open input file C:\Users\Frances\Downloads\quickbms\New folder\sharedassets1.assets.resS

- error in src\extra\xalloc.c line 619: xdbg_malloc()

Error: memory allocation problem
       Not enough storage is available to process this command.


press ENTER to quit

Here are the files I'm trying to extract:
https://mega.nz/#!FcMUVADT!oRTD-zgP-dJY ... fzLpUSwx5Q

Thanks!