Possible next features of QuickBMS

[aluigi]

QuickBMS 0.6.8 is almost ready so I'm trying to figure the small things to improve or fix.

Currently quickbms has tons of compression algorithms, encryptions and lots of file formats are covered so I would like to explore other usages rather than just extracting files.

Maybe a new experimental feature or a new input/output (sockets, processes and audio are already implemented from long time).

If you have ideas, even crazy ideas, feel free to propose them.

[aluigi]

What you think about the possibility of creating an ISO instead of extracting every single file?
I mean an option (for example -i) that converts the input archive in an iso that you can open and explore with any CD emulator or with imdisk.

[crushedice2000]

There is already a good file ripper. However I don't know if it will work on Windows.

http://binwalk.org/

[aluigi]

Just a news.
The ISO feature will be available in quickbms 0.7.1
archive.dat -> archive.iso

[aluigi]

In the week-end I will release QuickBMS 0.7.1.
Please let me know if you have any idea or feedback regarding something to fix/improve before this new version.

[aluigi]

One of the things I have to improve for sure is the output to the console because the Windows console is terribly slow.
If you take a script that generates lot of output (for example an archive with many files), try to run it on Windows and Linux to see the difference in times.
I guess the only solution will be to write a new console for the GUI mode because all the tests made on SetConsole*, setvbuf and so on are failing.

[aluigi]

Feel free to provide any feedback about the ISO feature of quickbms 0.7.1 (just released).

[aluigi]

The following are the features I'm evaluating for the next versions of quickbms:

• creation of a ZIP archive, just like the ISO feature. this is simple to implement and will be done in the next release.
• a configurable LZW engine like the one available in Camoto where you can set all the parameters
• a real GUI or something similar... mah
• a browser feature to view the files without extraction, there are some technical limitations I'm trying to solve: opening the selected file with the external viewers/editors requires to write the file on disk and requires to create a full GUI (directory tree visualization and options), using the memory requires to implement all the codecs and viewers inside quickbms and this will be NEVER done, a solution like a sort of virtual disk would be perfect but not possible in usermode without drivers to handle the IO. I'm thinking about something similar.

As usual, any idea, suggestion and feedback is welcome.

*edit* regarding the last point I was thinking to webdav but it writes the "remote" file to disk making it useless.

[einstein95]

Regarding that 2nd point, I was just going to ask if you'd include the LZW compressions used in Ultima 4 and Ultima 6, U4's uses a hash dictionary (decompressor: http://puu.sh/lor3r/910bd7d61a.zip, compressor: http://puu.sh/lor4N/880c791c94.zip) but I'm not sure what's different about U6's (decompressor: http://puu.sh/lor6p/49a0aa0f12.zip, compressor: http://puu.sh/lor6X/d70ce7dde8.zip)

[aluigi]

Ok, I will check them when I will implement the engine.

Another point I would like to add is the possibility of distributing a quickbms.exe with an embedded script in it.
The idea came from a request of an user and by some modkits I have seen around and I think that this feature would make their life and the one of the final users a lot easier.
Maybe a sort of feature activated if you add a certain resource or append the script to quickbms.exe.
It's easy to do so it costs nothing to me.

[aluigi]

I have just finished to add the output to ZIP that works like the ISO -i option but it creates a ZIP file.
There are also some new decompression algorithms:
- IBM TERSE (PACK and SPACK)
- PKWARE reduce (factor 1 -> 4)

As far as I know don't exist open source versions of these algorithms so it's something (hopefully) unique

[aluigi]

I have just finished to make the lzw engine of camoto/libgamearchive compatible with gcc (it doesn't like std::function).
The bad news is that the compressor is a dummy so it can't be used in reimport mode:
http://www.shikadi.net/camoto/doc/libga ... ml#details

In my tests I have some problems with the data generated by the dummy compressor but it may be caused by my modifications (bitstream replacement).
Currently I don't have an alternative ready to use but the code does its job in decompression so it's ok for me.
Maybe next time I will write my C library from scratch compatible with the initializers used in filter-lzw.cpp.

I have only to fix a small thing related to how are handled the last bytes in the stream (LZW_LEFTOVER_BYTES) and then it's ready.

[aluigi]

@einstein95
regarding U4 and U6, I guess that the difference is just in U4 that uses a fixed size codeword.
LZW has a codeword that starts from 9 bits and can reach a size of 16 while in U4 it starts as 12 and doesn't get modified.

I guess that U6 is just a classical LZW, anyway it will be available in quickbms 0.7.2 with its own comtype.

[aluigi]

Regarding the feature for the embedded script, I'm trying to find a solution that will work in any case and without forcing quickbms to parse its own executable.
I was thinking about the PE Checksum field but it requires to read the whole exe (quickbms is over 12Mb) and UPX sets it to zero making it useless.
Parsing the PE headers to check if the sum of all the sections match, may not work ever and it's still related to parse a part of the executable.
I was thinking about checking a constant string in the executable but it requires the modder to hex edit the executable.
Probably this feature will not be implemented, after all it's not necessary and will be used probably only in very few cases.

[aluigi]

Ok, I opted for the constant string that is 100% failsafe.
Basically the modder should:
- set one of the bytes of this hardcoded string in quickbms.exe to 0x00
- upx -9 quickbms.exe
- copy /b quickbms.exe script.bms output.exe

[aluigi]

QuickBMS 0.7.2 is planned to be released tomorrow.
Please let me know if there is a last-minute fix.

[shekofte]

browser feature !

i have a crazy idea on it
in first approach to your goal , how about to add a extraction report generator for quickbms ?
i mean by option of the user , quickbms instead of extraction of the archive , it just write a text file (may become compressed ) that contains useful information columns (name of file , size , offset ... ) about the content of archive !

for example , various members on web that extracted their own game archives just send (share) their extraction report between each other , so that if someone looking for the 3d model of "BMW car" or the .dds texture of a "sunflower" plant , can easily search through this text based report and explore for the desired archive ... more like an online contribution circle !
by this way we don't have to extract archives of all games on every computer .

[aluigi]

do you have some examples of such reports?
for example what's the most common format and what fields are used.

Anyway remember that sometimes quickbms works with memory files and chunked files, so a similar thing is totally useless.
That's also why the browser idea as been abandoned, because it's a process composed by 1 listing and N extractions that it's easy for various game formats but absolutely wrong for the others using custom obfuscations, chunked files and so on.
That's the problem of being a "generic" tool that must work in all the situations.

[shekofte]

excuse me sir for late !

for example what's the most common format and what fields are used.

when i looked more accurately , i found that the architecture of the report quite be flexible !
we can make the choices deeper and depend on the various file-types only useful fields that are keywords for that file-type will be printed ?
since it is possible by quickbms when an individual file decompressed on RAM just simply inspect the few bytes of header and extract the relevant fields ...

example for a .bmp or "BM" magic key according to this 010 editor binary template , the Width and Height Fields of image are in a fixed offset at 18 and 22 (decimal) :

http://www.sweetscape.com/010editor/templates/

Code: Select all

struct BITMAPFILEHEADER bmfh      0   14   Fg: Bg:0xE0E0E0   
CHAR bfType[2]   BM   0   2   Fg: Bg:0xE0E0E0   
DWORD bfSize   364662   2   4   Fg: Bg:0xE0E0E0   
WORD bfReserved1   0   6   2   Fg: Bg:0xE0E0E0   
WORD bfReserved2   0   8   2   Fg: Bg:0xE0E0E0   
DWORD bfOffBits   1078   10   4   Fg: Bg:0xE0E0E0   
struct BITMAPINFOHEADER bmih      14   40   Fg: Bg:0xE0E0E0   
DWORD biSize   40   14   4   Fg: Bg:0xE0E0E0   
LONG biWidth   607   18   4   Fg: Bg:0xE0E0E0   
LONG biHeight   598   22   4   Fg: Bg:0xE0E0E0   
WORD biPlanes   1   26   2   Fg: Bg:0xE0E0E0   
WORD biBitCount   8   28   2   Fg: Bg:0xE0E0E0   
DWORD biCompression   0   30   4   Fg: Bg:0xE0E0E0   
DWORD biSizeImage   0   34   4   Fg: Bg:0xE0E0E0   
LONG biXPelsPerMeter   0   38   4   Fg: Bg:0xE0E0E0   
LONG biYPelsPerMeter   0   42   4   Fg: Bg:0xE0E0E0   
DWORD biClrUsed   256   46   4   Fg: Bg:0xE0E0E0   
DWORD biClrImportant   0   50   4   Fg: Bg:0xE0E0E0   
struct RGBQUAD aColors[256]      54   1024   Fg: Bg:0xFFFFE0   
struct BITMAPLINE lines[598]      1078   363584   Fg: Bg:   

by this way i guess just by simple functions of quickbms can make such ideal reports that just need the offset of desired fields to predefined for each format (png ,wave . 3ds) or by identify of "MAGIC KEY" for quickbms !

may be the report looks something like this :
in the ascii text each line of it , is a record that belongs to a single file summary ! fields expressed in equation form

Code: Select all

"a0014.arc (ARCHIVE NAME)"
".bmp"   "sunflower(FILENAME)"      Width=607 Height=598 
.
.
.
".wav"   "clap (FILENAME)"       bitrate=44100 Channels=2
.
.
.
".3ds"      "BMW CAR(FILENAME)"       vertices=89 faceindices=136

[aluigi]

The only information that can be dumped are those you can see with -V
So:
- name of the variable
- offset of the field
- size of the field (like 2 for short)
- value read from that field

Maybe you mean a XML-like version of the -V output?