Possible next features of QuickBMS

[aluigi]

That thing of the memory files you suggested is complicated and without any real advantage.

URI/URL must use / and not \, that's part of the HTTP protocol.

[Shokoniraya]

anyway, it's really helpful to have such option in QuickBMS
(sorry, i wanted to change my previous comment but i write a clear one)
without any real advantage? names are more explicit than memory numbers

Code: Select all

set MEMORY_FILE1 string "1"
set MEMORY_FILE2 string "2"

math COUNT = 2
string MEMORY_SLOT p "MEMORY_FILE%s" COUNT
print "%MEMORY_SLOT%"
log memory_test1 0 1 MEMORY_SLOT #MEMORY_SLOT is MEMORY_FILE2, but it just acting as 0 file number, like this:(log memory_test1 0 1 0)

log memory_test2 0 1 MEMORY_FILE%COUNT% #not working, result must be 2, but output is 1, %COUNT% not working as MEMORY_FILE file number at all

and about names for MEMORY_FILE
result should be "take", but it does not using names for MEMORY_FILE
that means it's acting as unnumbered MEMORY_FILE (it's just a suggestion, using names for MEMORY_FILE could be useful)

Code: Select all

set MEMORY_FILE_take string "take"
set MEMORY_FILE_last string "last"

log memory_name 0 4 MEMORY_FILE_take #result should be "take"

[ExtractResponseUnit]

Thought about File Ripper

I catched up a few lines about to add such feature in QuickBMS. It's time expansive for sure and the solution would be a community-based
task. Unfortunately I'm not blessed with coding skills so I offer support in alternative ways. The wheel was already invented so I think it
would be less time consuming if you could implement a existing open source based file ripper and customize it instead making a neat one.

[aluigi]

@Shokoniraya
It would be difficult to implement that thing because all the files are just numbers in the bms language.
And the benefits are basically none.
You can just add a comment after each MEMORY_FILE to know what's meant for.

@ExtractResponseUnit
That would be nice if it's a separate tool than quickbms because it needs a different logic.
File rippers are difficult because sometimes there are no ways to calculate the size of the file, and scanning the magic values after the found one would produce many false positive.
I stay away from writing rippers

Instead, regarding the binary String feature implemented in the beta, I just noticed that it's not compatible with those situation where we have something like:

Code: Select all

getdstring PATH 0x20
getdstring NAME 0x20
string PATH + NAME

The result is just the original PATH and not PATHNAME because in binary mode the whole variables are concatenated, so the result is 0x20 bytes of PATH followed by 0x20 bytes of NAME.
Not sure what solution I'm going to implement but probably a parameter specified by the user for setting the binary mode.
Currently String has '0' as special prefix for the operator, what about using it also for the binary mode?
Like: string PATH 0+ NAME
Obviously the mode using NUL-delimited strings will remain the default as it is now.

[ExtractResponseUnit]

@ExtractResponseUnit
That would be nice if it's a separate tool than quickbms because it needs a different logic.
File rippers are difficult because sometimes there are no ways to calculate the size of the file, and scanning the magic values after the found one would produce many false positive.
I stay away from writing rippers

I don't know how to compile pascal source code but I think someone on this forum does so here we go . . .

[chrrox]

Does quickbms support these hash functions?
they are used in unity.
https://github.com/facebook/folly/tree/ ... folly/hash

referenced here
https://github.com/Unity-Technologies/U ... okyHash.cs

[aluigi]

Ok I just added spookyhash, murmurhash and xxhash in Encryption (not available in the online build yet).
Some of them are also available in the crc engine

[Shokoniraya]

sir aluigi, i found out that current beta version of QuickBMS has a problem

Code: Select all

log MEMORY_FILE6 0 0
#or even set MEMORY_FILE6 binary ""
put MY_VAR long MEMORY_FILE6

but it says: Error: MEMORY_FILE6 has not been used/declared yet
can't make a empty MEMORY_FILE?
there is no problem in 0.10.1 version

and one other problem, i've attached a bms file in this comment, QuickBMS can't download this file, i can download it from any downloader (it's a direct link), can you check it?
thank you

[aluigi]

Good catch on that MEMORY_FILE problem, I'm checking it right now.

I tried passing download/file.php?id=9258 to quickbms 0.11 as input file and it correctly downloads it, are you talking to some different problem?

[Shokoniraya]

yes, i'm takling about some direct links that can't download it with QuickBMS

please download attached file in my previous comment (link_bms.zip) and open zip file, then use test_link.bms
it's a bms file to download a direct link, but can't download it with QuickBMS by using that script

i'm talking about test_link.bms file (bms open "." ""), not attached zip link

[aluigi]

Just checked and the behavrious of quickbms is correct.
In short your link is a huge file of 2147483648 bytes, so quickbms tries to allocate such amount in memory for downloading it and fails.

The maximum amount you can allocate is 2147221471 bytes (Content-Length field) which is crazy because quickbms is NOT a downloader.
It will not tell you if the file exists but it will first download it completely.

[aluigi]

I just updated the beta on the website.
I did some changes to the multidimensional arrays (VAR[i] and VAR[i][j]) so please check them too.

[Shokoniraya]

about huge files, what about 4gb version? has same problem since it should handle unit

i tested array in some different ways, looks like works fine, can you tell some detail about those changes?

[aluigi]

Not sure if 4gb bypasses it but for sure you should not use quickbms for downloading 2Gb files

Regarding the changes, they are mainly programming stuff without additional feature.

One change was that now VAR[] is considered a stand-alone variable itself while before it was a multidimensional array with a "" index that basically is just the same as now.
My plan is that maybe I can use it as a sort of way to receive the total number of entries or just leave it as-is.

I also moved a function that allocates memory for the arrays (add_multi_dimensional) because it was leaking lot of RAM for nothing.
Basically just this but I'm sure it may be possible to optimize it even more.

[Shokoniraya]

i think you should leave it be same as is (sorry for my limit grammar)
and just a question, can you check if it's possible to optimize read/write function, making it faster?

and one more thing, some of files are read-only, i know it can simply disabled, but can you do something about Error: Permission denied? (passing read-only, i mean if input file was read-only, then quickbms disable read-only for that file)

[aluigi]

What you mean with making faster read/write?
Do you have an example?

Good idea the "read-only to write" thing, going to add it soon.

[aluigi]

Ah that permission denied is NOT related to read-only files.
It's related to the permissions of the folder, long story short I will never touch that.
But I did the read-only thing when using -w, in that case quickbms showed the message: Do you want to create it from scratch (y/N)?

[Shokoniraya]

sir aluigi, i found a problem in savepos, i know it can be done with a little more coding in bms file. but can you check it? i think savepos has problem in asize (reimport mode)

i've attached a zip file in this comment
there is a savepos.bms in zip file, put quickbms.exe in savepos_problem folder, and goto file_content folder and open a file with name sec_3 in hex editor, and add a few byte (or remove a few byte) from that file
and click on _import.bat, and you will see that END_OFFSET (in bms file) not changed in reimporting (asize problem in savepos)

also, there is same problem with xmath and math too, becuase they will not change in reimporting

can you check other commands that related to such cases too?

[aluigi]

There was for sure a problem with asize because I fixed it few hours ago while checking another topic in which clog added a 32bit field into the compressed data (but I don't think it's the problem you refer).

Regarding reimport3 and savepos... it's a mess.
quickbms already resets the variables containing the asize value without considering any math operation on them.
For savepos it would be complicated to adjust them.

[aluigi]

New beta is available:
http://aluigi.org/beta/quickbms_beta.zip

(the beta is complete: full source code and both the exe compiled for release)