Possible next features of QuickBMS

[aluigi]

math name_size * 2

[tbmq008]

yeah... that
i added that line of code before the getdstring line and still, this is what i get

Code: Select all

. 0000000000000058 getdstr base_path  "." 20
    2e 00 2e 00 2f 00 2e 00 2e 00 2f 00 2e 00 2e 00   ..../...../.....
    2f 00 00 00        

[aluigi]

That's correct.
I don't think I understand what other output you need, read the raw data and convert it with "set base_path unicode base_path".
That's how it works and in fact it perfectly works in my UE4 script.

[tbmq008]

yeah, i got this.

[chrrox]

add these hash functions?
I did not see them listed in quickbms txt.
https://sites.google.com/site/murmurhash/
https://github.com/PeterScott/murmur3

[aluigi]

the beta already has them.
if they are not listed in Encryption hash, take a look at crc.c:

Code: Select all

         if(ctx->type == 0)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ MYCRC) & 0xff] ^ (MYCRC >> 8))
    else if(ctx->type == 1)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ (MYCRC >> (ctx->bits - 8))) & 0xff] ^ (MYCRC << 8))
    else if(ctx->type == 2)  CRC_CALC_CYCLE(    ((MYCRC << 8) | MYBYTE) ^ ctx->table[(MYCRC >> (ctx->bits - 8)) & 0xff])
    else if(ctx->type == 3)  CRC_CALC_CYCLE(    ((MYCRC >> 1) + ((MYCRC & 1) << (ctx->bits - 1))) + MYBYTE)
    else if(ctx->type == 4)  crc =              crc_in_cksum(MYCRC, data, datalen);
    else if(ctx->type == 5)  CRC_CALC_CYCLE(    MYCRC ^ MYBYTE)
    else if(ctx->type == 6)  CRC_CALC_CYCLE(    MYCRC + MYBYTE)    // lose lose
    else if(ctx->type == 7)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ MYCRC) & 0xff] ^ MYCRC)
    else if(ctx->type == 8)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ MYCRC) & 0xff] ^ (MYCRC >> (ctx->bits - 8)))
    else if(ctx->type == 9)  CRC_CALC_CYCLE(    (MYCRC << 1)  ^ MYBYTE)
    else if(ctx->type == 10) CRC_CALC_CYCLE(    (MYCRC << 1)  + MYBYTE)
    else if(ctx->type == 11) CRC_CALC_CYCLE(    rol(MYCRC, 1, 0) ^ MYBYTE)
    else if(ctx->type == 12) CRC_CALC_CYCLE(    rol(MYCRC, 1, 0) + MYBYTE)
    else if(ctx->type == 13) CRC_CALC_CYCLE(    ror(MYCRC, 1, 0) ^ MYBYTE)
    else if(ctx->type == 14) CRC_CALC_CYCLE(    ror(MYCRC, 1, 0) + MYBYTE)
    else if(ctx->type == 15) CRC_CALC_CYCLE(    (MYCRC << 5) + MYCRC + MYBYTE) // djb2 5381
    else if(ctx->type == 16) CRC_CALC_CYCLE(    (MYCRC * ctx->poly) + MYBYTE) // djb2 and sdbm
    else if(ctx->type == 17) CRC_CALC_CYCLE(    (MYCRC * ctx->poly) ^ MYBYTE) // djb2 and FNV-1
    else if(ctx->type == 18) CRC_CALC_CYCLE(    (MYCRC ^ MYBYTE) * ctx->poly) // FNV-1a
    else if(ctx->type == 19) CRC_CALC_CYCLE(    MYBYTE + (MYCRC << 6) + (MYCRC << 16) - MYCRC) // sdbm 65599
    else if(ctx->type == 20) CRC_CALC_CYCLE(    ctx->poly * (MYCRC + MYBYTE * (i + 1)))
    else if(ctx->type == 21) crc =              qhashmurmur3_32(data, datalen);
    else if(ctx->type == 22) crc =              qhashfnv1_32(data, datalen);
    else if(ctx->type == 23) crc =              qhashfnv1_64(data, datalen);
    else if(ctx->type == 24) crc =              XXH32(data, datalen, ctx->poly);
    else if(ctx->type == 25) crc =              XXH64(data, datalen, ctx->poly);
    else if(ctx->type == 26) crc =              jenkins_one_at_a_time_hash(data, datalen);
    else if(ctx->type == 27) crc =              xPear16(data, datalen);
    else if(ctx->type == 28) crc =              CityHash32(data, datalen);
    else if(ctx->type == 29) crc =              CityHash64(data, datalen);
    else if(ctx->type == 30) crc =              CityHash64WithSeed(data, datalen, ctx->poly);
    else if(ctx->type == 31) crc =              StormHash(data, datalen, MPQ_HASH_TABLE_INDEX);
    else if(ctx->type == 32) crc =              StormHash(data, datalen, MPQ_HASH_NAME_A);
    else if(ctx->type == 33) crc =              StormHash(data, datalen, MPQ_HASH_NAME_B);
    else if(ctx->type == 34) crc =              StormHash(data, datalen, MPQ_HASH_FILE_KEY);
    else if(ctx->type == 35) crc =              StormHash(data, datalen, MPQ_HASH_KEY2_MIX);
    else if(ctx->type == 36) crc =              jenkins_hashlittle(data, datalen, ctx->poly);
    else if(ctx->type == 37) crc =              adler32(0, data, datalen);
    else if(ctx->type == 38) crc =              fnv32(data, datalen, crc ? crc : 0x811c9dc5);
    else if(ctx->type == 39) crc =              UHash(data, datalen, crc, ctx->poly, 0x7fffffff);
    else if(ctx->type == 40) crc =              spookyhash_32(data, datalen, ctx->poly);
    else if(ctx->type == 41) crc =              spookyhash_64(data, datalen, ctx->poly);
    else if(ctx->type == 42) crc =              XXH3_64bits(data, datalen);
    else if(ctx->type == 43) crc =              XXH3_64bits_withSeed(data, datalen, ctx->poly);

[aluigi]

QuickBMS 0.11 is finally out
http://quickbms.com

[shekofte]

QuickBMS 0.11 is finally out

It is gold

[Shokoniraya]

really out! thank you, sir aluigi!
i wish many good times for you, sir aluigi

can i give you a suggestion? can you do something about pickle as a data-type?
for example:

Code: Select all

get MY_VAR pickle

i know it's stupid and blineded for users as a var type (based on what i found in pickle.bms), but useful

maybe a guessvar needed for that

Code: Select all

get MY_VAR pickle
guessvar MY_VAR_TYPE MY_VAR
print "%MY_VAR_TYP%"

[aluigi]

It's a bit difficult to implement it because pickle is a mix of various types of basic and complex serializers, that's why it's great as bms script but difficult to "embed" in quickbms (for various technical and design reasons).
pickle is also rarely used.
Isn't the script-library good enough in doing the job?

[Shokoniraya]

It's a bit difficult to implement it because pickle is a mix of various types of basic and complex serializers, that's why it's great as bms script but difficult to "embed" in quickbms (for various technical and design reasons).
pickle is also rarely used.
Isn't the script-library good enough in doing the job?

i know that, and also it can't used in put or some other things
but useful to passing serialized bytes or reading blocks and reverse it back in reimport-mode
scripting is good enough, but a var type could be handy

[Ekey]

RAD Game Tools got bought by Epic Games, and they've made Oodle at least available on GitHub: https://www.unrealengine.com/en-US/blog ... via-github

Oodle source is included in three zip files, located in

Code: Select all

Engine/Plugins/Compression/OodleData/Sdks/2.9.0/src/oodle2_src_for_unreal_data_2.9.0.zip
Engine/Plugins/Developer/TextureFormatOodle/Sdks/2.9.0/src/oodle2_src_for_unreal_texture_2.9.0.zip
Engine/Plugins/Compression/OodleNetwork/Sdks/2.9.0/src/oodle2_src_for_unreal_network_2.9.0.zip

maybe useful

[aluigi]

License is still a limitation, indeed the source code is not public.
But at least we will see many more oo2*.dll in the future games that we can embed in our tools.
The old oodle dll currently used in quickbms will still be mandatory since some algorithms are not compatible (or no longer available) in newer versions of oodle.
Definitely a good news btw

[Shokoniraya]

hello sir alugi, just want to say that append + put* not working (tested on last release)

(same example by you)

Code: Select all

set MEMORY_FILE binary "0123456789"
goto 4 MEMORY_FILE
append
putdstring "hello" 3 MEMORY_FILE
append
get SIZE asize MEMORY_FILE
log "output.dat" 0 SIZE MEMORY_FILE

[aluigi]

It depends by what you want to do:
append: put it at the end
append 1: replace the data at offset 4
append 2: put it at offset 4 making space in the file (like reimport3)
append -1: at the beginning

Everything seems to work as intended

[aluigi]

A note for who uses the String command in quickbms 0.11 with the following operators (lower case): b, e, c.
I had to update some of my scripts because the way quickbms 0.11 works will return different results than 0.10.1.

It's a sort of bug because in theory it should be forced to use the length of the variable but instead it handles it as a NUL delimited string.
I guess I made the mistake while working on the String command and I used the length of the variable in a first moment and later switched to NUL delimited forgetting to fix b/c/e.
In short b/e/c give the same result of B/C/E and you must use the '0' prefix to make it work properly.

I will definitely fix it in the next version but who knows when it will ever be released.
Solution: String VAR1 0b VAR2

The solution is compatible with old versions of quickbms too

[tbmq008]

here is a feature request from me.
generate an entire tar file instead of extracting every file.
if quickbms can generate an entire ISO and zip file why not tar? all the files inside a tar are uncompressed, and a user may have to deal with millions of files (including duplicate ones, or files that look like such judging by their names or even files with weird names) anyway.

[aluigi]

tar is much more simple to implement than ISO and ZIP, it doesn't even have a header or footer.
It's so simple that it can be done just with some bms code around the log/clog command in the script used to dump the files.
(The only limitation would be with output filenames longer than 99 bytes since it requires more code for "././@LongLink")

I'm not sure how many users may need support for a tar output, I don't even know if anyone uses the ISO and ZIP outputs

Just curious, why did you think to tar?
Isn't ISO better due to the mounting on any operating system?
And why do you have millions of output files? That sounds exagerated.

[tbmq008]

Just curious, why did you think to tar?
Isn't ISO better due to the mounting on any operating system?
And why do you have millions of output files? That sounds exagerated.

because it's easy to see whatever's in there in a hex editor, but also so i don't have to deal with compressed stuff (zip).
not sure about mounting though.
so i can do something with them? honestly i don't have that many files lying around but... yeah.

just saying it would be nice to have a "tar" feature.

[aluigi]

Yeah mounting works perfectly.
I just tried it now for confirming that it works correctly (for being 100% sure about the current version) by using zip.bms on the zip of quickbms source code with the -i option, I mounted the generated ISO and I had full access to all the "extracted" files.

Adding an additional output like tar would be like opening a pandora's box for me, because if I can add a new output format I would like to add N new formats maybe even with possible external non-builtin formats via user's dll or some definition/script code.

Currently adding only tar doesn't give any benefits to users compared to ISO and ZIP.

If you need to check a large amount of files with hex editor, one option may be to use a binary grep for searching a specific string (I wrote a tool called mygrep), or maybe a preview of the first bytes of each file (like "quickbms hexdump_scanner.bms folder"), or using the -O FILE option of quickbms that dumps all the files in one unique file called FILE.
Just some ideas.