Possible next features of QuickBMS

[michalss]

QuickBMS 0.9.0 is finally out:
http://quickbms.com

Where is a download link luigi pls ? Is exe same as dll ?

[aluigi]

The link is not on the homepage but I posted it some posts ago and it's in quickbms.txt:
http://aluigi.org/papers/quickbms_dll.zip

[michalss]

Having hard time to make it work on C# to be honest..

Code: Select all

[DllImport("quickbms.dll", EntryPoint = "quickbms_compression")]
        private static extern int quickbms_compression(string algo, byte[] inData, int zsize, byte[] outData, int size);

        private void testCompression()
        {
            var rr = quickbms_compression("", null, 1000, null, 5000);

        }

Not sure if this is correct, dont know how to properly declarate it and how to write decompress and compress functions...

[michalss]

Hi Aluigi, we have finally found some time and implemented small wrapper around your great quickbms dll tool for C# and it works great for decompression and encryption / decryption. The only problem we now have is with compression and hashes - zlib_compress or deflate_compress etc. does not work like they should ( - error in src\extra\xalloc.c line 678: xdbg_realloc()
Error: memory allocation problem), we have also tested NULL array as output data, which eliminates error, but produces no comprimed data (returned size is zero). As for hashes, we have no idea how to get the computed hash - so far it returns identical data as input. If currently not possible / implemented with hashes -
we suggest another separate interface for hashes with custom hash output (instead of replacing input) and hash_size (in byte form is fine) is an option or replace input completely by resizing input array to default hash size (after computation - but this option is least favored)

[aluigi]

ok for compression I understand what's the problem, it's exactly what it says so a problem during allocation because the compression "reallocates" the buffers which is something impossible with the buffers provided externally from your tool.
The quickbms_hash variable can't be read from the dll.
Added to my TODO list and will check what I can do in the next release, consider that (re)compression and hashing are not exactly the features I was thinking to for the dll, I made it mainly for the decompression and encryption that you confirmed work correctly.

[michalss]

ok for compression I understand what's the problem, it's exactly what it says so a problem during allocation because the compression "reallocates" the buffers which is something impossible with the buffers provided externally from your tool.
The quickbms_hash variable can't be read from the dll.
Added to my TODO list and will check what I can do in the next release, consider that (re)compression and hashing are not exactly the features I was thinking to for the dll, I made it mainly for the decompression and encryption that you confirmed work correctly.

Nice thx for info and btw great job. Main reason to do this is to use compression and decompression and encryption in own tools, so compress is very important in this case as well.

[AnonBaiter]

Code: Select all

putarray 0 0 -0.534796
putarray 1 0 -2.33745618
putarray 2 0 -1.77319
putarray 3 1 0.199748
putarray 4 1 1.900874
putarray 5 1 2.44347639
...

let's just assume i'm going to write a bms script that needs this many "arrays"(about 70 ARRAY-s per 20 VAR_IDX-es each, or 1 ARRAY per 80 VAR_IDX-es each if you will) to even do anything - surely i'm not going to write it like this
point is, i need a simplified putarray command so that having to come up with gigantic arrays stored within a bunch of indexes doesn't turn out to be an exercise in patience and frustration

[aluigi]

Do you mean something like the following?

putarray 0 -1 VALUE
putarray 0 -1 VALUE
putarray 0 -1 VALUE
...
putarray 1 -1 VALUE
putarray 1 -1 VALUE
...
sortarray 0 1 # the last 1 will automatically applying the sorting to all the arrays and not just to array 0

[AnonBaiter]

kinda like that yaa

although what i had in mind was more like this

Code: Select all

putarray 0-2 0 -0.534796 -2.33745618 -1.77319
putarray 3-5 1 0.199748 1.900874 2.44347639

[aluigi]

way too complex

[AnonBaiter]

its actually pretty simple to me

[aluigi]

if something is completely different than the expected syntax (PutArray ARRAY VAR_IDX VAR) then it can't be simple at all.

The only space available is after the 3rd argument which is reserved for future usages, anyway quickbms supports a limited number of arguments (a bit less than 32).

[shekofte]

I kiss your hands Luigi Auriemma for all of your efforts !
Providing of quickbms as dll must be a new generation !
I ashamed to say you even in my current age i am not properly familiar with programming of codes that insert dll commands !!!
I begging for a very very simple tutorial may be in c++ that show how to use quickbms DLL ?

[aluigi]

I posted simple C examples few posts above:
viewtopic.php?p=35965#p35965

[shekofte]

I posted simple C examples few posts above:
viewtopic.php?p=35965#p35965

acknowledge

[michalss]

Hi Aluigi, is there any ETA to fix compression and encryption issues in DLL pls ? Tomb raider is comming and we would like to use it for that.. Thx a lot

[aluigi]

Honestly I still don't have a date for the next version of quickbms (and consequently quickbms.dll) but I expect it for the end of September.

Currently that specific problem with the dll is not very simple to handle because the internal functions of quickbms, like the compression/encryption ones, are meant to work with the memory of quickbms while in that dll the memory is provided from outside.

What I mean is that it would be impossible to support some comtypes like unzip_dynamic of lzma_dynamic but I can do 2 things to improve the compatibility:
- disabling the memory protection of quickbms (basically the same that happens when you run quickbms.exe with the -9 option, I didn't check if this is already default)
- avoid reallocating memory when quickbms_compression and quickbms_encryption are used

[michalss]

Honestly I still don't have a date for the next version of quickbms (and consequently quickbms.dll) but I expect it for the end of September.

Currently that specific problem with the dll is not very simple to handle because the internal functions of quickbms, like the compression/encryption ones, are meant to work with the memory of quickbms while in that dll the memory is provided from outside.

What I mean is that it would be impossible to support some comtypes like unzip_dynamic of lzma_dynamic but I can do 2 things to improve the compatibility:
- disabling the memory protection of quickbms (basically the same that happens when you run quickbms.exe with the -9 option, I didn't check if this is already default)
- avoid reallocating memory when quickbms_compression and quickbms_encryption are used

Well im sure it is not simple, but im 100% convidence you will manage it as always.. Thx keep us posted..

[aluigi]

In the next version quickbms will have an option (-U) for listing all the supported compression algorithms.
And when using the comtype scanner it will dump the files with the name of the algorithm instead of their number, yeah no longer annoying navigation if defs.h for locating the algorithm.

[aluigi]

Since reimport2 is not limited by the size of the files, I can implement as many fake (aka nop) compressors I desire.

For example some compression algorithms are available only in their decompression implementation lacking the code necessary for recompressing the data, therefore reimporting is not possible.
A fake compressor would bypass the problem.

If interested please let me know what algorithms you would like to see implemented.
Even better provide the code for fake compressor yourself