Possible next features of QuickBMS

[AnonBaiter]

Okay, now I`m getting the gist of it.
However, the result was like this(reconstructed header):

Code: Select all

00000000  52 58 57 53 20 B4 01 00 00 02 00 00 00 00 00 00  RXWS ´..........
00000010  46 4F 52 4D 20 00 00 00 00 01 00 00 00 00 00 00  FORM ...........
00000020  01 00 00 00 02 1C 03 00 7F 7F 00 00 00 01 C0 5D  ..............À]
00000030  00 00 00 00 00 00 00 00 C0 B3 01 00 00 00 00 00  ........À³......
00000040  46 54 58 54 20 00 00 00 00 01 00 00 00 00 00 00  FTXT ...........
00000050  01 00 00 00 08 00 00 00 61 6D 30 30 30 32 00 00  ........am0002..
00000060  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000070  00 00 00 00 C0 B3 01 00 00 01 00 00 00 00 00 00  ....À³..........
00000080  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000090  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000A0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000B0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000C0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000D0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000E0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000000F0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000100  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000110  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000120  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000130  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000140  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000150  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00000160  00 00 00 00 00 00 00 00 00                       .........

The header I`m trying to make looks like this(original header):

Code: Select all

00000000  52 58 57 53 30 D5 01 00 00 02 00 00 00 00 00 00  RXWS0Õ..........
00000010  46 4F 52 4D 20 00 00 00 00 01 00 00 00 00 00 00  FORM ...........
00000020  01 00 00 00 02 1C 03 00 7F 7F 00 00 00 01 C0 5D  ..............À]
00000030  00 00 00 00 00 00 00 00 44 BF 09 00 00 00 00 00  ........D¿......
00000040  46 54 58 54 20 00 00 00 00 01 00 00 00 00 00 00  FTXT ...........
00000050  01 00 00 00 08 00 00 00 74 6F 30 30 36 34 5F 6B  ........to0064_k
00000060  6F 75 65 6E 5F 31 00 00 00 00 00 00 00 00 00 00  ouen_1..........
00000070  42 4F 44 59 C0 D4 01 00 00 01 00 00 00 00 00 00  BODYÀÔ..........

So, is there any way to construct the header after that part without leaving unwanted blanks(0x00) like the original RXWS header shown above?

[AnonBaiter]

Sorry for the double post, but I`m here to report a thing.

Seems that apparently comtype_scan2.bat doesn`t work with me anymore.
I mean, when I execute the .bat file through command-line... it just doesn`t run on folders that has symbols and numbers. And that`s on Windows XP.

Code: Select all

1. Type this and press enter.
C:\Documents and Settings\AnonBaiter\Desktop\quickbms_0.8.0>comtype_scan2.bat
2. Even when the .bat file in question is executed(and even when there are parameters involved(comtype_scan2.bat comtype_scan2.bms dump.dat c:\output_folder)), it does nothing.
C:\Documents and Settings\AnonBaiter\Desktop\quickbms_0.8.0>

Any solutions for this?

[MerlinSVK]

Does this work?

Code: Select all

"C:\Documents and Settings\AnonBaiter\Desktop\quickbms_0.8.0>comtype_scan2.bat"

[AnonBaiter]

I guess that solved the problem. Thanks.

[chrrox]

Does quickbms support these hash functions?
https://github.com/aappleby/smhasher
Horizon zero dawn is using this one
https://github.com/aappleby/smhasher/bl ... rHash3.cpp

[aluigi]

I think yes but configuring it may be not easy, the following is probably ok for 32bit murmur:

Code: Select all

encryption crc "" "32 0 0 21"
get SIZE asize
log MEMORY_FILE 0 SIZE
print "%QUICKBMS_CRC|x%"

[shekofte]

Hello sir Luigi Auriemma
every moment contact with you is so honorable for me
after all dear Luigi i come with a crazy offer !
i hope some kind of confirmation from you ?
adding a new function or element that i called it DOSWINDOW
i want to say the DOSWINDOW is an advanced case of print command !
in fact a command that allow users to open as many dos windows they need linked to quickbms !
in other words these windows will work very similar to your MEMORY_FILEs or TEMPORARY_FILEs with the exception they are smaller and are manipulative by users for versatile purposes !
i believe this concept will increase interactivity of qickbms exponentially !
making of them are so simple and just a few functions of like printf , scanf ...
the advantage of doswindows is that they can easily arranged by operating system itself and may be they can be colorful also as you know the full screen of a doswindows can be copy pasted as text ...
...
for example user can make a window for hex view of a portion of an archive in a specific offset !
windows that list all variables used by quickbms and show their value or their content ! (debugging)
or another window for manual editing cells of an array !
as well as the process of re importing Archives can become more customizable !
i guess there is many benefit ?
even the bms script itself can opened in doswindow (colored code) pluse a separate window that contains listed bms functins that they will inserted just by pressing enter button in the code window and another window for list of compression algorithms and one for list of encryptions ...
and in the end i have a dream to write bms script while qickbms attached to the archive and by adding each line of code at the same time i see the value of the variable or when i define an offset i see the hex section of that archive offset ...
============================
DEAR Luigi Auriemma i apologize if my idea looks foolish and the verbose text cause headache .

[aluigi]

Many time ago I had in mind to implement some step-by-step debugging feature for showing information like the current content of the variables and the content of the input file at the current offset in some windows.
The idea was abandoned because the time and effort for implementing it is not worth the real advantages it gives.
For example for me (I mean for the personal use I make of quickbms) it's enough the -V option together with the "less" tool, never had the need of additional debugging in all these years

[Acewell]

hi, can you add this marmoset mview decompression to quickbms?
http://forum.xentax.com/viewtopic.php?p=119352#p119352

[aluigi]

Sure.

[Acewell]

hi, can you add this marmoset mview decompression to quickbms?
http://forum.xentax.com/viewtopic.php?p=119352#p119352

Sure.

what is the name you gave this compression, i need to update my mview script.

[aluigi]

artstation
I didn't have samples to test it so let me know if it works correctly

[Acewell]

thanks
so far it worked on the samples i tried

[BSM]

I'm new to QuickBMS, and while I have tried to search the forum for my doubt, I have found only a single reference to dictionaries (as pairs of key + value, being keys unique, and where you can retrieve values given a key) as a question from chrrox in this thread: http://zenhax.com/viewtopic.php?p=3370#p3370

I tried to create a bms script to handle .obsp files to help Zeleboba as per this post: https://zenhax.com/viewtopic.php?f=9&t=6927, but the file structure includes a dictionary of 64 bits keys + string values in a way that certain keys are used several times in the files content, and supposedly must be replaced by the corresponding strings. I know that such a complex file format maybe not the best point to start coding bms scripts, but I'm used to coding in several languages and I think I understand how to create bms scripts; it is just that the needed dictionary seems a handicap... And that is needed also for another file formats for the same game (The Guild 3).
How would you approach that?

I actually don't need to use QuickBMS for this, as I already coded my own tool for this file format, but I'm curious and maybe it could be handy for another file formats in future, for my own use and/or to share it.

[aluigi]

How would you approach that?

I wouldn't since that's not the job of quickbms
Extracting data from a binary file: ok.
Handling json, strings and other textual formats: not (or at least not easily and 100% compatible).

If I need a simple dictionary of keys and values I would just use arrays like:

Code: Select all

putarray 0 -1 key1
putarray 1 -1 value1
putarray 0 -1 key2
putarray 1 -1 value2
...
putarray 0 -1 keyN
putarray 1 -1 valueN

And then using getarray accordingly but it's just a waste of time, searching a specific key many times will be very slow.

In the example you provided then you don't have just a list of keys and values, you have a full structure with nested elements (json) which is a completely different thing.
*edit* sorry I was referring to your example meant as json output handled as input.

You don't need to create key/value pair to spit out a json output from those obsp files, it's enough to parse them with a nested function and printing the output in json format like you did.
It's very easy to be honest, I do that many times with some archive formats

[BSM]

How would you approach that?

I wouldn't since that's not the job of quickbms
Extracting data from a binary file: ok.
Handling json, strings and other textual formats: not (or at least not easily and 100% compatible).

I guessed it, but thanks anyway for the answer.

In the example you provided then you don't have just a list of keys and values, you have a full structure with nested elements (json) which is a completely different thing.

Well, in fact the dictionary of strings contains both the filenames and the strings used inside each file, so if just unpacking to extract the files in binary form, it could be done without using the real filenames and full subfolders path, if not using the dictionary. For another file packages (.oppc), the content is a bunch of related meshes, textures and descriptors that forms a single or a multiple 3D element, so just the descriptors uses some nested properties (like json), but the different parts can be assembled into 3D objects by reading into that nested properties how all parts are related (the walls, the roof, etc, with their respective coordinates and other properties).
Also the keys are some kind of 64 bits hash (same string in different files uses the same key, so I guess it is a hash), but with an unknown algorithm; unknown for me, it is nothing I have tried, including any type of Checksum, so I am unable to pack different files, if not using same names (by using the same hashes).
Anyway, I coded a tool in C#, able to unpack and then even to translate the unpacked files into human readable text files (not polished the last part, as there are at least 2 different byte codes to produce different types of arrays, but at least to give an idea).

[aluigi]

With quickbms you can easily handle nested content if there are no references outside the current tree.
For example archives with a nested TOC are very easy to extract with quickbms.

[episoder]

lemme do a feauture request. for those sorta files i write headers for. is it possible to flag a log from a memory file (ouput header assembly) as dummy for reimporting? example...

Code: Select all

dmlog outname 0 128 memory_file ; file header assembly
append
log outname data datasize 0

means it can write the header the regular way and extract and apped data seperate. when reimporting the dmlog header is skipped and only the data is re-inserted.

[aluigi]

Yeah good idea.
I solved the problem by simply allowing the -. option to do the job without additional commands.
That option is used for continuing the execution of quickbms in case of errors so it fits the case since MEMORY_FILE reimporting is not possible and this is a way to "bypass" it.
This feature will be available in quickbms 0.8.2 that I will release in the next days.

[aluigi]

Ok guys, do you remember the idea of an alternative reimporting mode that appends the new file at the end of the archive and rewrites the fields containing offset, size and possible compressed size?
Quickbms 0.8.2 will have this new mode available as experimental by using -r twice like -r -r

It was very simple to implement this mode by adding few lines of source code and worked immediately in my tests.
That doesn't mean it will work in ALL the situations due to the obvious limitations of the various formats existent but if there are no MEMORY_FILEs involved and no operations on the offset/size/zsize fields (so "math OFFSET * 0x800" will not work), it should work correctly.