Possible next features of QuickBMS

[aluigi]

Currently in my todo list there are the following entries:

• automatic parsing and decompression of the xcompress files, those with the \x0f\xf5\x12\xee signature.
  They are rarely used and additionally it seems to exist some variants (WatchDogs) with additional 2 short fields... boring
• a way to save and restore the global endianess, it would be useful for file formats with platform-related endianess and containing files that use a specific endian to restore after the process

My dream feature would be the formats ripper but it's really too expensive in terms of work and effort.
Basically I would like to add in Quickbms a way to recognize many known file formats (like PNG, DDS, Ogg and so on) and extracting them "blindly" on the fly.

What is a file ripper?
Take a file with unknown content, scan it and dumps all the known files you can recognize.

The reason is that I'm not aware of real supported and "trusted" file rippers available.
The existent ones are jurassik or completely vanished (mripper and jauber naub) or of unknown status (aka old) or quite suspicious.

So it's NECESSARY to create a new tool to do this work that seems easy but it's quite complex because you must understand many file formats of various types.

Maybe it's even possible to create a new stand-alone tool but I think that QuickBMS already has all the code to do the job.

Let me know what you think or if you know other alternatives, and also if you have suggestions for fixes and improvements of QuickBMS.

[aluigi]

The next release will have many bug fixes.
I'm finding many issues in the last days like the auto-extension features on filename (not ""), some memory consumptions, a problem with multiple getarray and so on.

[michalss]

I guess you should really focus on packing archives, this is most important for me for example. Extract various files with know extention is nice but not important as packing features to be honest.

[aluigi]

Quickbms already has the reimport feature that works with many scripts.

The alternative is writing custom packers specific for each game, and so not something related to quickbms.

[cyanic]

I think QuickBMS can be used for writing files from scratch; I've seen a couple of scripts do that. I think the biggest thing that can be improved is pluggable encryption/decompression routines. One of the reasons I don't really want to write QuickBMS scripts is the lack of support for newly discovered encryption/decompression routines, which means it's faster for me to knock up something in C# than to submit code, wait for it to be included, then write script and tell everyone they need to upgrade their QuickBMS.

[aluigi]

And what you think about the dll to use embedded in the script with the CallDLL command?
It works also with the dumped functions.

I use that feature often for small custom encryptions because, after all, the most known encryption algorithms are already implemented in quickbms.

[cyanic]

Interesting, when was that implemented? Does it allow loading of external DLLs? What about calling conventions?

[aluigi]

It was implemented the 17 Oct 2009 in version 0.3.4.

It supports tons of calling conventions, you can see them in the src file calling_conventions.h:
usercall, cdecl, stdcall, thiscall, msfastcall, borland, pascal, watcom, safecall, syscall, optilink and clarion.

And yes, you can download also external libraries.
Personally I love the possibility to load dumped functions like those simple decryption routines that stay in one single function without static arrays.

Obviously, for security reasons, the user will be informed of the loading and he must agree to run the code (except if he uses the -C option).

[cyanic]

Cool, did not know that.

[RetroHelix]

Here are my thoughts about a fileripper.
I only know Jaeder Naub. The ui is a bit untidy but I like that you can tick every setting easily. If you are going the commandline way it should be possible to scan for an exact filetype (jpg, gif...) and for a group of files e.g. "graphics" instead of only one filetype (to save me from having too much to type ) You could manage groups of filetypes in an ini file or something. It would also be good to have the option to output a result list without ripping anything. I would also like to see a kind of definition file that holds magicnumbers or fileheaders and can be easily expanded. Mmmhh, I don't know how useful some kind of "deep scan" (scanning archives inside archives) would be...

[aluigi]

Currently what stops me to make a file ripper is mainly related to the supported formats and how allowing the users to expand these "rules".
For example I can implement PNG, BMP and RIFF in C language inside the tool.
Then an user asks me to add support for a model format like MDL3, and so I need to understand the format and what are the best ways to:

• identify the format
• parse the format
• be sure at > 90% that what I'm dumping is an MDL3 file, so all check routines
• be prepared to possible new versions of the format or small modifications of which I wasn't aware

It's not a simple job.

The solution would be something community-based where the work is not on my shoulders and the tool can be expanded in a very easy way.
Something like making a text file with the rules of identification, parsing and dumping. Put these sort of textual plugins in a folder and you have a complete ripper.

So probably the idea would be something based on Python but:

• you need to know Python
• you need to have Python
• it would be a tool with many small tools (python scripts)
• far from the idea of the text rules I had in mind

So there are many difficulties but I believe that a ripper is necessary, I can't tell people to use a file ripper for some strange archives if there are no rippers or serious tools available.

[RetroHelix]

Why not use the BMS script language for this?

[aluigi]

Because it's not much flexible and is slow.

I thought about this idea but image scanning a file of 2 gigabytes with 1000 scripts that operates on every byte to check the signature.
Let's say there is a solution for the scanning, the it may be difficult to parse the data from a script.

For example the TGA file are usually recognized from their last signature, in that case there is the risk that the script reads the data owned by the previous ripped file.
The script should be easy so without 1000 checks.

Anyway it's not impossible, with some format it would be trivial:

Code: Select all

savepos OFFSET
idstring "RIFF"
get SIZE long
math SIZE += 8
log "" OFFSET SIZE

[aluigi]

New version of QuickBMS out: 0.6

Changelog:

• review and better writing of quickbms.txt
• source code extension H to C
• -a can be specified multiple times for new arguments
• new argument for the Append command to choose when and how write the new data
• added new names for String operators
• Endian can save the current endianess in a variable
• Get line works even if there is no new line delimiter at end of file
• last offset visualized when a compression fails
• additional arguments for String printf
• fix for base64_compress that wasn't handled
• added comtype lz4_compress
• new behavior of Debug command
• base_offset argument for Padding command
• negative index variable for get/putarray for taking and storing elements at the end of array
• a positive value in goto SEEK_END is automatically converted in negative to seek correctly
• fix for the filenames ending with . or * to guess their extension
• better colors for the hexhtml visualization
• automatic handling of xcompressed files (\x0F\xF5\x12\xEE) in comtype xmemcompress
• some malloc to calloc changes
• new icon
• changed major version number due to the huge amount of enhancements from 0.5

[Ekey]

Good work!

[aluigi]

What you think about a possible feature for dumping strings, that you can edit with a text editor, and then reinjecting them back in the file?
Something similar to the current reimport feature, but specific for strings.

Is there are real improvement compared to the manual "hex editing" method?

[lavrov]

A couple of small bugs, or, rather, nuisances I stumbled upon debugging my scripts.

1) https://github.com/artlavrov/nmotools

Strings built using the same code look different (note the reverse slash):

Code: Select all

...
182352       32 objects\00000999_0x00000418_PARAMETERIN(2) "FadeIn/Out"
182384       68 objects/00001000_0x00000419_BEHAVIOR(8) "GotoScene"
...

Both are constructed using string NAME p= "objects/%08d_0x%08x_%s(%d)" i C_ID TYPE C_TYPE
I have no idea why it happens exactly when i==1000.

Tried to export images to tga using this code, is there another (faster) way?

Code: Select all

         # planar to RGBA conversion, really slow
         for k = 0 < PLANE_SIZE
            set OFS k
            for p = 0 < PLANES
               goto OFS MEMORY_FILE5
               get C byte MEMORY_FILE5
               put C byte MEMORY_FILE4
               math OFS += PLANE_SIZE
               next p
            next k

Also are there any plans on built-in conversion to png?

2) https://github.com/artlavrov/paktools

There is a possibility to use Quickbms along with Far Manager's MultiArc plugin (see custom.ini).
Quickbms-powered interactive MultiArc feels and looks just great (give it a try).
But in order to work, quickbms should be able to unpack files without paths.
Probably it can be done with ARG1, but it would need to change all scripts.
I tried to specify a single output file "quickbms -f <filename> -O <filename_without_path>".
Finds out quickbms still creates (empty) directory trees, probably it's not needed with this option.

Upd: also maybe it's possible to add a "less verbose", quiet option (-q?), that would only list files without copyright and coverage lines (frankly, I need it because MultiArc can't handle stderr and stdout separately and doesn't support redirection).

[aluigi]

Hi lavrov.

Regarding the first point I'm not able to replicate the problem.
I even used your nmo.bms script with the file provided by another user and modified it to go over 1000 files but nothing.

For the tga there are no alternatives, quickbms is a file extractor not a programming language
For this type of operations I usually write a small function in C, I compile it, I dump the function and then I use it inside the script.
The only downside is that the user is prompted before running the function.

For point 2, yes you must use the -O option.
The creation of folders while using -O is a bug, I will fix it in the next version.

[aluigi]

To anyone:
I have almost finished to implement and fix all the things I planned for version 0.6.1.
So if you have other features to suggest, that's the right moment

To lavrov:
I have read your update just now (a new post was a better idea), I can check if I may add a -Q option or just do it with the already existent -R one but honestly I don't care, the quickbms title is already sent to stderr so the fault is that multiarc.
In addition, why someone would use that multiarc if quickbms is already complete and stand-alone?
This is not a ZIP or specific format supported by a tool, it's a scripting tool that is not bound to a specific format.
Honestly I don't understand it.

[TGE]

Would it be possible for you to add a wildcard function when using for example:

Code: Select all

findloc offset string "N*IF"

With * representing the wildcard?