Possible next features of QuickBMS

[aluigi]

I just released a new beta of quickbms with experimental support for TAR when using the -O option, example:
quickbms -O output.tar script.bms archive.dat

There is no support for long filenames since it was just an experimental feature consisting of a few additional lines of code.
Basically I can easily add support to any format that uses metadata and sequential content.

Please remember that -O appends data to the existing output.tar because it's an option meant for concatenating the files even when launching quickbms on a folder with multiple input files.

This beta also fixes a bug with callfunction when using multidimensional arrays (VAR[x]), a bug when using encryption calldll with append mode (quickbms asked to overwrite the file for no reason), a bug in the size field of encryption incremental, plus other stuff fixed in the previous betas.

The link is ever the same of all the previous beta versions:
https://aluigi.altervista.org/beta/quickbms_exe.zip

[spiritovod]

a bug when using encryption calldll with append mode (quickbms asked to overwrite the file for no reason)

Oh, so it was a bug after all. I wonder how you've found out that issue.

Anyway, could you please add some additional info about using RSA encryption to documentation. I failed to understand how it works in quickbms context. At least, I couldn't get it somehow working with keys in binary format (like from this topic) or PEM format (with public/private key headers) or converted versions of it (PEM format consists of modulus and exponent as last 4 symbols, if I got it right). I mean, quickbms simply crashed or produced zero output. Comparing to some known functions like openssl_public_decrypt or respective analogs from wincrypt / CryptoPP would be also appreciated.
Thanks in advance.

[aluigi]

I spotted that bug while fixing the size field of the "incremental xor" encryption, due to a bug in the size (32bit was wrongly interpreted) I had to rely on a tcc function, since only the first part of the file was encrypted I had to use "encryption calldll + log + encryption "" + append + log" and that's how I noticed something wrong

Regarding the private key encryption... it's a mess.
There are various different implementations and libraries so there is definitely a way in quickbms but I don't know how since it depends by what implementation you want to emulate.

I leave you some topics that may be useful:
viewtopic.php?t=13947
viewtopic.php?t=14660

This is a brief explanation of what API are called for each Encryption command that may be related to private key operations:

rsa tomcrypt:
only the key is used

rsa:
the key is given to PEM_read_bio_* for all the supported methods available in openssl (RSA_PUBKEY, RSAPublicKey, RSAPrivateKey, X509, SSL_SESSION and so on) plus BN_bin2bn (for 'n' a sort of raw public key encryption like modpow) if everything else fails
the ivec is only used with BN_bin2bn for the 'e' field so it's ignored in normal conditions.

modpow:
as above, key for 'n' and ivec for 'e'.
It's also possilble to set the 'zed' field of modpow if specified (example: Encryption modpow_zed)

PKCS5_PBKDF2_HMAC or PKCS5_PBKDF2_HMAC_SHA1 and so on:
it calls "PKCS5_PBKDF2_HMAC(key, keysz, ivec, ivecsz, cycles" where cycles is 1000 by default or any number you prefer if you set the encryption mode, example: Encryption PKCS5_PBKDF2_HMAC key ivec 1234

Rfc2898DeriveBytes:
the first part works just like PKCS5_PBKDF2_HMAC plus it performs an AES 256 CBC encryption.
It uses sha1 by default like for the previous one, if you need other algorithms try: Encryption Rfc2898DeriveBytes_MD5 ...

BytesToKey:
Example:
Encryption BytesToKey KEY IVEC 1000
Encryption "BytesToKey aes_256_cbc sha1" KEY IVEC 1234
It's a raw call to EVP_BytesToKey using aes 256 cbc, sha1, and 1000 cycles by default

PBKDF1 / PBKDF2
It's a wrapper for tomcrypt, also in this case KEY, IVEC, CYCLES.
It's also possible to specify the hashing algorithm and if we want to use openssl for PBKDF1:
Encryption "PBKDF1 SHA1 openssl" KEY IVEC 1000


It's a mess, yeah I fully agree!
Tons of libraries, different ways for doing the same thing in other API and libraries.

[spiritovod]

Sorry, it seems it's not quite clear about what I'm doing. Please open an online RSA decryption/encryption and key generator tool and try to decrypt any random data with any given valid public key (or private key, not sure). I suppose RSA is data-independent, so you should be able to somehow "transform" any data as long as you're using valid key format. Also you can take a look at the topic, mentioned in my previous post - binary key there looks like ASN.1 type of RSAPublicKey.
The problem is that I couldn't use or reproduce any of this in quickbms, or at least get any result from encryption rsa/rsa_tomcrypt, except for crashes or zero output.

[aluigi]

It's openssl that refuses to load the key, there is nothing I can do.

The code for loading the key is simple and it's also confirmed on all the examples on the Internet:

Code: Select all

keybio = BIO_new_mem_buf(key, keysz);
if(keybio) {
    rsa = PEM_read_bio_##X(keybio, NULL, NULL, NULL);

where ##X is a cycle of the following methods:

Code: Select all

DHparams
DSAPrivateKey
DSA_PUBKEY
DSAparams
ECPKParameters
ECPrivateKey
EC_PUBKEY
NETSCAPE_CERT_SEQUENCE
PKCS7
PKCS8
PKCS8_PRIV_KEY_INFO
PUBKEY
PrivateKey
RSAPrivateKey
RSAPublicKey
RSA_PUBKEY
SSL_SESSION
X509
X509_AUX
X509_CERT_PAIR
X509_CRL
X509_REQ

Doesn't matter if the key is binary, base64, between "-----BEGIN ... -----END...", everytime that function returns NULL.
I even tried passing rsa as second argument but it's still the same.

*edit* even replacing BIO_new_mem_buf with BIO_new(BIO_s_mem())+BIO_write doesn't work (just checked since the former may need a const key valid till the end of the decryption, which is not this case btw)

If you find a way to load the key with openssl I will immediately implement it.

[aluigi]

Ok it works.
It's openssl which is a disaster in handling the input key.

Long story short openssl wants the key in base64 encapsulated between "-----BEGIN PUBLIC KEY-----" and "-----END PUBLIC KEY-----".
ADDITIONALLY it also wants the key to be truncated at the 64th character so you can't copy&paste a long key but you must ensure that each line is not "too long".

So if you use an RSA encryptor online you can decrypt it with quickbms.
I will work on the beta for making all those "manual" changes automatically made by quickbms.

[spiritovod]

Thank you for looking into the issue. I've tried before with passing "set KEY string "public key with proper tabs and header" " as key, but it didn't work, that's why I've asked about particular format. Do input should be base64 encoded as well for "encryption rsa KEY" to work as expected in this case?

[aluigi]

KEY:

Code: Select all

-----BEGIN PUBLIC KEY-----
line1_64_chars
line2_64_chars
line3_64_chars
...
lineN_64_chars
-----END PUBLIC KEY-----

Input file must be binary.

Example script:

Code: Select all

set KEY string "
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----
"

get SIZE asize
encryption "rsa" KEY
log "output.dat" 0 SIZE

Probably you will notice that the output has the same size of the input, that's because quickbms ignores any returned size from the encryption commands since they gave some problems in the past with some algorithms.

[aluigi]

Since you linked that topic of atom0s I made a script for using that key in quickbms:

set KEY string "
30 81 89 02 81 81 00 E2 72 5E F9 BB 16 88 71 C2 38 D9 1B 64 CF B8 B1 33 2F 1B BC F1 05 F4 0F 25 2F B9 3F 3A 60 9D 52 4C F8 F5 EE 09 BC 55 4F D9 18 DB 8B B3 53 1D 6F 88 BE FE A4 BF BD F5 1C B1 E1 DF 5E 5D FA 83 FD 65 84 D3 7E 27 99 24 22 4F C4 F8 BB 6C 98 ED 50 D2 70 02 E8 BA 21 F3 5F 01 55 A0 8D 9E D2 76 71 40 32 AE EC DA 06 6C 17 FA 54 F1 C3 3E 5D AF 8B 33 2B 3C C0 77 14 90 A1 52 61 B2 DD 90 8F 53 F1 02 03 01 00 01
"
string KEY h KEY
comtype base64_compress
string TMP_KEY 0c KEY

strlen TMP_KEYSZ TMP_KEY
set KEY ""
math KEYSZ = 0
for x = 0 < TMP_KEYSZ
getvarchr TMP TMP_KEY x
if x % 64
elif x != 0
putvarchr KEY KEYSZ '\n'
math KEYSZ + 1
endif
putvarchr KEY KEYSZ TMP
math KEYSZ + 1
next x

# the following don't work?
#string KEY p "%s\n%s\n%s\n" "-----BEGIN PUBLIC KEY-----" KEY "-----END PUBLIC KEY-----"
#string KEY P "-----BEGIN PUBLIC KEY-----\n%KEY%\n-----END PUBLIC KEY-----\n"

string KEY p "%s%c%s%c%s%c" "-----BEGIN PUBLIC KEY-----" '\n' KEY '\n' "-----END PUBLIC KEY-----" '\n'

print "%KEY%"

get SIZE asize
encryption "rsa" KEY
log "output.dat" 0 SIZE

Now my doubt is about those two commented "string KEY p/P" because I don't understand why the \n escape doesn't work.
I will check it later to know if it's something that needs to be fixed.
The Print command handles the escape, while String P doesn't.

*update*
Regarding String VAR p the problem is that the format argument is given as-is to sprintf without handling the escapes, I can fix it but I hope it will not make some scripts incompatible, for example: string NAME p "%s\%s PATH NAME
I ever use / so I'm worried about the scripts of other users.

[spiritovod]

Great, thanks for the script, it will allow to handle binary rsa keys as well. Alternatively, you can also convert them manually with any hex->base64 converter, add header and tabulate string to 64 char parts. Also a small note to myself, this is block encryption and data should be aligned to 16 bytes and key size in general, otherwise function will fail.

Update: Nice, after undestanding general key format and block thing with aligning, I managed to decrypt actual data with private key by manually converting it like explained above and using it on properly managed blocks. For private keys "BEGIN/END RSA PRIVATE KEY" header should be used. Thanks again for the help.

Update 2: Though I'm curious why do you need an empty string at the beginning and at the end of "string" variable. It's the same with C code for tcc, if there is no empty string at the start, code compilation will fail.

[aluigi]

I guess there was a discussion years ago about the "format" of the multi-line strings but no idea when and where it was.

Instead regarding that supposed bug in "String p/P" I think I reached a compromise:

- "p": if I use printf I expect that the format string contains escaped characters like in programming, but at the same time in quickbms that argument is a string... so there is no good or wrong.
A work-around may be to handle it as an escaped string if it's used directly in the command and as a string if it's a variable, example which should have the same result:
escaped: string VAR p "%s\n%s\n" VAR1 VAR2
normal string: set FMT binary "%s\n%s\n" ; string VAR p FMT VAR1 VAR2

- "P": same as above, when I use it I expect that the output of String P is identical to Print, also in this case the work-around would be like the same.

I have already implemented these work-arounds in the current beta on the website (https://aluigi.altervista.org/beta/quickbms_exe.zip) and it finally works as expected.
I look forward for any comments about it, hopefully it would have limited impact on existing scripts except for those using \ for example as path delimiters in building filenames.

The beta also contains all the automated code for building a valid key for openssl, no matters if the BEGIN/END ar missing or if it's directly the binary code because quickbms will do the job itself.

Additionally there is also a new feature that I was testing: ImpType.
This was a command of the original MexScript that was totally useless in quickbms due to the automatic work of the reimport mode.
With the new modes of ImpType is now possible to write an updated variable in the original field in any reimport mode.
From quickbms.txt:

Code: Select all

        imptype var ORIGINAL_VAR NEW_VALUE
            put NEW_VALUE at the location of ORIGINAL_VAR field

        imptype var ORIGINAL_VAR
            update the field where ORIGINAL_VAR was read with its
            current value

        imptype off -16 NEW_VALUE
            put NEW_VALUE at the offset 16 bytes at the end of the
            file, its size is the same of the field that has been
            read at that location or a 32bit LONG if never read

        imptype off 16 NEW_VALUE
            as above but it's at offset 16

        imptype crc ORIGINAL_VAR
            replace the field of ORIGINAL_VAR with the current
            content of QUICKBMS_CRC or QUICKBMS_HASH, it requires
            the usage of Encryption with the relative crc/hash
            algorithm used in the *Log operation

Obviously I'm open to comments and improvements since this is just a test I tried and it worked well (needs more testings obviously).
I think it has a big potential for finally solving the problems of the crc and hash fields used in some archives.

*edit* I fixed the examples of imptype above because the "mode" was missing

[aluigi]

Another new beta is out.

I fully implemented the ImpType CRC method:

Code: Select all

        imptype crc ORIGINAL_VAR
            replace the field of ORIGINAL_VAR with the current
            content of QUICKBMS_CRC or QUICKBMS_HASH, it requires
            the usage of Encryption with the relative crc/hash
            algorithm used in the *Log operation
            this command must be used AFTER *Log

        imptype crc ORIGINAL_VAR ALGO ...
            replace the field of ORIGINAL_VAR with the crc or hash
            calculated during a *Log operation and based on the
            chosen ALGO and its parameters, SAME as Encryption.
            this command must be used BEFORE *Log.
            the crc/hash is calculated on the data before
            encryption and before compression
            Example:
                get MYCRC long          # read the field
                imptype crc MYCRC md5 ""
                comtype zlib
                encryption xor "mykey"  # compatible with imptype
                clog NAME OFFSET ZSIZE SIZE
                imptype crc ""          # optional disable

I did some tests and it worked very well.
I also fixed few other things and I had to make some changes to core functions like perform_encryption for fully implementing ImpType CRC, so I expect some bugs.

With this feature it's now possible to bypass all the limitations caused by crc and hash in archives but we need more tests.

[aluigi]

New beta and new feature plus various bug fixes.

The new feature is related to calldll tcc and the file I/O, since it wasn't supported I replaced it with memory I/O.
Long story short any call to API like fgetc/fputc/fread/fwrite will use the input and output provided in calldll, it's enough to pass the argument as &#INPUT# and &#OUTPUT#.

Example:

Code: Select all

set MEMORY_FILE10 string "

int func1(FILE *fd) {
    int     i, c;
    for(i = 0;; i++) {
        c = fgetc(fd);
        if(c < 0) break;
    }
    return i;
}

int func2(FILE *fd, FILE *fdo) {
    int     i, c;
    for(i = 0;; i++) {
        c = fgetc(fd);
        if(c < 0) break;
        c ^= ' ';
        fputc(c, fdo);
    }
    return i;
}
"

get SIZE asize

encryption calldll "MEMORY_FILE10 func1 tcc RET &#INPUT#"
log MEMORY_FILE 0 SIZE
encryption "" ""
print "%RET%"

comtype calldll "MEMORY_FILE10 func2 tcc RET &#INPUT# &#OUTPUT#"
clog MEMORY_FILE 0 SIZE SIZE
print "%RET%"

And yes, now any MEMORY_FILE, #INPUT#, #INPUT_SIZE#, #OUTPUT#, #OUTPUT_SIZE# can be passed as pointers which may be useful in some rare situations.

[Ekey]

https://github.com/widberg/fmtk/wiki/As ... ompression

[Shokoniraya]

sometimes need to append MEMORY_FILE to input file and that is alright, but there should be a mlog, just like normal log but it can accept importing binary from MEMORY_FILE to file too
i know most of scripts will broken if do such thing but it can be usefull for manually jobs so no need to create a file if we want to replace a part of file

example:

Code: Select all

set MEMORY_FILE1 binary "DATA"
get MEM_SIZE1 asize MEMORY_FILE1
reimport 3
mlog MEMORY_FILE1 0 MEM_SIZE1

[Shokoniraya]

hello sir aluigi, thank you for releasing QuickBMS version 12
can you please add these 4 things in your todo list if you want to?

1- when opening a folder to do operation, please make a possibility to keep some variable's value in next run, or even MEMORY_FILE data not going to lost instead of have to saving it to a file for next time that a file will opened up. for usage of (quickbms.exe -Y -o -Q -. -F "{}.bin" "Folder" "Output_DIR")

2- add an print option to not break in with 0x0A [\n]. for example (xprint "Exporting %FILE_NAME%...") just like printf("");

3- is -Q are full silent for all of QuickBMS functions? if there is still a few error or report prints like version of QuickBMS at the top, please make it full silent for all of them

4- automatically rename a file or folder to add a number at end of name if it is not accessible or writable instead of request enter a new name if process must be done on many files and not to have check it always for such cases

[deymomc]

New beta and new feature plus various bug fixes.

The new feature is related to calldll tcc and the file I/O, since it wasn't supported I replaced it with memory I/O.
Long story short any call to API like fgetc/fputc/fread/fwrite will use the input and output provided in calldll, it's enough to pass the argument as &#INPUT# and &#OUTPUT#.

Example:

Code: Select all

set MEMORY_FILE10 string "

int func1(FILE *fd) {
    int     i, c;
    for(i = 0;; i++) {
        c = fgetc(fd);
        if(c < 0) break;
    }
    return i;
}

int func2(FILE *fd, FILE *fdo) {
    int     i, c;
    for(i = 0;; i++) {
        c = fgetc(fd);
        if(c < 0) break;
        c ^= ' ';
        fputc(c, fdo);
    }
    return i;
}
"

get SIZE asize

encryption calldll "MEMORY_FILE10 func1 tcc RET &#INPUT#"
log MEMORY_FILE 0 SIZE
encryption "" ""
print "%RET%"

comtype calldll "MEMORY_FILE10 func2 tcc RET &#INPUT# &#OUTPUT#"
clog MEMORY_FILE 0 SIZE SIZE
print "%RET%"

And yes, now any MEMORY_FILE, #INPUT#, #INPUT_SIZE#, #OUTPUT#, #OUTPUT_SIZE# can be passed as pointers which may be useful in some rare situations.

I rented the new version of QuickBms can you extract the latest version of the game https://www.zuixiaoyao.com/download.htm?

[Shokoniraya]

hello sir aluigi, can you please check this two problem?

1) network problem
i dont know if QuickBMS using native os connection or any other libraries such as Curl, but please use wget2
it is not like old wget, it is so much better than other libraries
https://gitlab.com/gnuwget/wget2

2) folder checking
a command really needed to check if a folder exist or not, some of archives have same name files and folders
for example:
UI\maps\SP_01\PauseMenu_01.pvr
UI\maps\SP_01

it must rename manually, but if i want to do it like this, i can not make any progress, because i can't check if it exists or not (to able to add PHASE_ to name string if something happens in if able to check folders or files):
note: adding PHASE_ is a part of bms script, not a requested feature, like (string PATH = "PHASE_01\\" ; string PATH + NAME)
PHASE_01\UI\maps\SP_01\PauseMenu_01.pvr
PHASE_02\UI\maps\SP_01

command should check if a folder or file exist with same name and return 0 or 1
example: DirGet FDSE DIR_NAME EXIST_VALUE #check folder or file exist or not
also too: DirSet FDSE DIR_NAME MODE #create empty folder if MODE is 1, else, create blank file if MODE is not set or any other number

[Ekey]

Some compression variants

https://github.com/lab313ru/lzkn
https://github.com/lab313ru/megapack-megadrive
https://github.com/lab313ru/xperts_cmp
https://github.com/lab313ru/rnc_propack_source
https://github.com/lab313ru/itl_comp
https://github.com/lab313ru/brave_prove_slid
https://github.com/lab313ru/tf3compression
https://github.com/lab313ru/desert_unpacker
https://github.com/lab313ru/LZCapTsu
https://github.com/lab313ru/lztoshio