QuickBMS header and other info through error stream

[FatalBulletHit]

It seems like QuickBMS uses the error stream for the program header and other information, while the standard stream only contains log info, here's an example:

Error Stream

Code: Select all

NotSpecified: (:String) [], RemoteException
QuickBMS generic files extractor and reimporter 0.10.1
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Oct 20 2019 - 14:53:23)
NotSpecified: (:) [], RemoteException
                          quickbms.com  Homepage
                            zenhax.com  ZenHAX Forum
                     @zenhax @quickbms  Twitter & Scripts
NotSpecified: (:) [], RemoteException
- open input file .\sounds_eng_014457777.pck
- open script .\wwise_pck_extractor_customized.bms
- c_structs (0): "return"
- set output folder .
NotSpecified: (:) [], RemoteException
  offset   filesize   filename
--------------------------------------
NotSpecified: (:) [], RemoteException
- 28 files found in 0 seconds

Standard Stream

Code: Select all

  00001000 453115     english(us)\sounds_eng_014457777\0x79a34bee.bnk
  00070000 1700716    english(us)\sounds_eng_014457777\0xc7088efa.bnk
  00210000 672968     english(us)\sounds_eng_014457777\0xf3fd406e.bnk
  002b5000 28171      english(us)\sounds_eng_014457777\0x7b30ec05.wem
  002bc000 15359      english(us)\sounds_eng_014457777\0x7b30ec1a.wem
  002c0000 18769      english(us)\sounds_eng_014457777\0x7b30ec35.wem
  002c5000 35923      english(us)\sounds_eng_014457777\0x7b31eb52.wem
  002ce000 25950      english(us)\sounds_eng_014457777\0x7b32ecc5.wem
  002d5000 30845      english(us)\sounds_eng_014457777\0x7b34ece9.wem
  002dd000 31139      english(us)\sounds_eng_014457777\0x7b35ed68.wem
  002e5000 35541      english(us)\sounds_eng_014457777\0x7b36ed3e.wem
  002ee000 23876      english(us)\sounds_eng_014457777\0x7b37ebd3.wem
  002f4000 30825      english(us)\sounds_eng_014457777\0x7b38ec9c.wem
  002fc000 14414      english(us)\sounds_eng_014457777\0x7b38ec9e.wem
  00300000 38945      english(us)\sounds_eng_014457777\0x7b38eca7.wem
  0030a000 29144      english(us)\sounds_eng_014457777\0x7b39ed03.wem
  00312000 25852      english(us)\sounds_eng_014457777\0x7b41ed2d.wem
  00319000 25214      english(us)\sounds_eng_014457777\0x7b42eb20.wem
  00320000 15458      english(us)\sounds_eng_014457777\0x7b42eb40.wem
  00324000 19177      english(us)\sounds_eng_014457777\0x7b43ea87.wem
  00329000 18041      english(us)\sounds_eng_014457777\0x7b43eace.wem
  0032e000 20900      english(us)\sounds_eng_014457777\0x7b44e9af.wem
  00334000 16507      english(us)\sounds_eng_014457777\0x7b44e9f2.wem
  00339000 15128      english(us)\sounds_eng_014457777\0x7b45ec61.wem
  0033d000 27925      english(us)\sounds_eng_014457777\0x7b46ea33.wem
  00344000 21630      english(us)\sounds_eng_014457777\0x7b46ea54.wem
  0034a000 18377      english(us)\sounds_eng_014457777\0x7b46ea55.wem
  0034f000 26437      english(us)\sounds_eng_014457777\0x7b46ea64.wem

I use PowerShell to call QuickBMS, in the attachment is everything you need (except QuickBMS itself), to reproduce the error simply extract the archive, copy quickbms.exe into the folder and run reproduce_error_stream.ps1 with PowerShell (right click on file).

reproduce_error_stream.zip

I've also noticed, that the error stream contains multiple NotSpecified: (:String) [], RemoteException errors which seem to be connected to running QuickBMS in PowerShell (https://stackoverflow.com/a/2095623/9248774).

[spiritovod]

Aside from the way quickbms handles errors, I'm curious, what you're trying to achieve by using quickbms in a powershell script like this.
For example, "return" statement doesn't exist in quickbms syntax and produces c_structs error. Also, in the powershell script you're executing the same code twice (4th and 11 lines), so maybe files or memory page are being blocked somehow and the exception is being raised. If you need to debug bms script, it's better to use build-in verbose options like -v or -V. Or, if you want to execute something in batch, you can use -F (for specific files) or slightly modify that bms script to control output.

[FatalBulletHit]

Thanks for the input, but I wrote this script simply for visualization of the issue and from what I can tell the error lies not within the PowerShell script nor the QuickBMS script (are you sure return doesn't exist? In the example for "CallDLL DLLNAME" it is used).
Also, QuickBMS doesn't use the full potential of the CPU, so running it parallel allows for decreased extraction duration.
I've also barely any idea of the QuickBMS language, so extracting files to variable directories would not only take longer to extract but would also require me to learn the QuickBMS language.

[spiritovod]

CallDLL invokes C/lua/python code and compiles it on the fly. So yes, "return" doesn't exist in bms syntax, but can be used in С or other code, integrated into bms script (which is not the case here). As for parallels, I doubt it will work with quickbms if it's not designed for it (and in most cases extraction logic requires sequential execution - I mean not multiple files extraction, but memory allocation).
The reason why I've suggested modifing bms script is because currently it's using offsets as filenames - and while you're extracting all files in a single folder, there is non-zero chance that offsets (and thus filenames) will overlap in some particular cases.

[aluigi]

Yes, quickbms uses two separate streams for separating the important output (stdout/info) from the verbose information for the user (stderr).

I don't know why Powershell gives that "NotSpecified: (:) [], RemoteException" with empty lines, quickbms just uses 'fprintf(stderr, "\n...")' without even setting 'setbuf(stderr, NULL)', so just the basic usage.

[aluigi]

Ah, there is no multi-threading because all the operations are sequential and need to be completed before going to the next offset/information (it would be also terrible for the I/O input other than impossible to program).

[FatalBulletHit]

Thanks for the quick replies! :)

CallDLL invokes C/lua/python code and compiles it on the fly. So yes, "return" doesn't exist in bms syntax, but can be used in С or other code, integrated into bms script (which is not the case here).

Alright, good to know, it's not my script, I'll create an issue on their GitHub page. ^^

As for parallels, I doubt it will work with quickbms if it's not designed for it (and in most cases extraction logic requires sequential execution - I mean not multiple files extraction, but memory allocation).

Ah, there is no multi-threading because all the operations are sequential and need to be completed before going to the next offset/information (it would be also terrible for the I/O input other than impossible to program).

Well, I honestly only understand half of the stuff you're saying and I think I worded it badly, but having e.g. 4 instances of QuickBMS run with each extracting a different file can be significantly faster than having one instance extract all 4 files.

The reason why I've suggested modifing bms script is because currently it's using offsets as filenames - and while you're extracting all files in a single folder, there is non-zero chance that offsets (and thus filenames) will overlap in some particular cases.

I'm using the auto rename switch, should that not take care of it or am I'm misunderstanding something here?

Yes, quickbms uses two separate streams for separating the important output (stdout/info) from the verbose information for the user (stderr).

Correct me if I'm wrong, but shouldn't stderr only contain something when there was an error? Either way, I'll just workaround the issue by checking if stderr contains the string "error" and if it does, I'll log it completely, otherwise I'll ignore it.

[spiritovod]

I'm using the auto rename switch, should that not take care of it or am I'm misunderstanding something here?

If it's fine for you, I guess there is no problem then.

[aluigi]

Well, I honestly only understand half of the stuff you're saying and I think I worded it badly, but having e.g. 4 instances of QuickBMS run with each extracting a different file can be significantly faster than having one instance extract all 4 files.

Ah ok so you are referring to the directory as input which runs the recursive file scanning.
Yes I guess it would be faster but I see some headaches for me and the users:

• the users would probably be scared of seeing many cmd windows opening at the same time
• that would also affect the focus of Windows because the user can't do other things while the new dialogs spawn
• an alternative would be to assign a list of files to each one of the 4 instances of quickbms but I'm not sure how easy it would be to prgram it
• it needs some additional programming from my side, if there is any little mistake there is the risk of the user being flooded with endless spawning processes of quickbms

Correct me if I'm wrong, but shouldn't stderr only contain something when there was an error? Either way, I'll just workaround the issue by checking if stderr contains the string "error" and if it does, I'll log it completely, otherwise I'll ignore it.

It's up to the programmer I guess.
Splitting the output between stderr and stdout it's great for programs that act as wrapper of quickbms.exe and for who wants to debug without having the banner and other information visible.