Possible next features of QuickBMS

[aluigi]

The fake compressors are a huge success.
I already implemented most of those that were missing in my scripts and they all work:

Code: Select all

DRV3_SRD_COMPRESS
YAZ0_COMPRESS
BIZARRE_COMPRESS
BIZARRE_SKIP_COMPRESS
BLACKDESERT_COMPRESS
DR12AE_COMPRESS
EA_COMP_COMPRESS
LBALZSS_COMPRESS
MOHLZSS_COMPRESS
MOHRLE_COMPRESS
NISLZS_COMPRESS
QCMP1_COMPRESS
RFPK_COMPRESS
RLEW_COMPRESS
SAINT_SEYA_COMPRESS
SHREK_COMPRESS
SLZ_01_COMPRESS
SLZ_02_COMPRESS
SLZ_03_COMPRESS
UCLPACK_COMPRESS
SEGA_LZS2_COMPRESS
WOLF_COMPRESS
YAKUZA_COMPRESS
YKCMP_COMPRESS

Implementing them is really a joke requiring one minute if you have the source code of the decompression algorithm and no bit-operations are involved.

[Shokoniraya]

looks like QuickBMS has issue with single files in reimport2
can you just add a tag to replace raw file (not re-overwrite OR write in next data)
i know its a problem in a single file without OFFSET, but a command-option must disable this error and replace raw file, not write. just for single files that no offset are located

if you do that, then we can even make scrip for text and string files!

i really cant reimport2 such files!

[aluigi]

In that case you should use reimport.bat and then select "force" when quickbms asks you if you want to continue due to your file being bigger than the original.

There is no issue or bug, reimport.bat and reimport2.bat work exactly as they are meant to do

[Shokoniraya]

In that case you should use reimport.bat and then select "force" when quickbms asks you if you want to continue due to your file being bigger than the original.

There is no issue or bug, reimport.bat and reimport2.bat work exactly as they are meant to do

i know about force mode, quick bms cant updata file size in force mode and reimport2 cant import bigger file in whit this script
my QuickBMS vesion; 0.9.0 (2018-17 jun)

[aluigi]

You don't need quickbms to do something like that and in any case you are doing it in the wrong way because it's not just a single file that you can put back ignoring all the data that was available after it and the other fields referring to sizes and offsets.
You have at least 2 size fields at offset 0x90 and 0x94 and the data from offset 0x351b is NOT the ogg file, it's part of the archive format/container that you are going to screw by overwriting them.
It's extremely important to never overwrite data that doesn't belong to the extracted file and why reimport and reimport2 work in that way.

[Shokoniraya]

You don't need quickbms to do something like that and in any case you are doing it in the wrong way because it's not just a single file that you can put back ignoring all the data that was available after it and the other fields referring to sizes and offsets.
You have at least 2 size fields at offset 0x90 and 0x94 and the data from offset 0x351b is NOT the ogg file, it's part of the archive format/container that you are going to screw by overwriting them.
It's extremely important to never overwrite data that doesn't belong to the extracted file and why reimport and reimport2 work in that way.

Aluigi, then READ THIS

in this case, plase read this note carefully

i want to say some things, real and important. im gonna tell you some Property that Must add into QuickBMS.exe

1_Importing In Bigger Size;
as you can see, quick bms is verry good for anyone, who wants starting to editing files and archive. Admittedly, need To Improt Bigger Size in Files
im not tlking about reimport2 for set new offset and new size! im talking about memory_size or those files without offset!

2_Multi Size:
in my Previous post, my file had multi size,
i cant see any tag in QuickBMS.exe for update both size.
some archve has Archive_Size, like UMD (Unreal Engine 2.5) and for import2, Archive_size must turn to read base size!
also some archives has decompressed_size and compressed_size, and we cant do it with Quick bms, and thats not well for someone like me that cant make a new tool or dont know any other programing language Like C# or C++ and even C.
why QuickBMS? (for any other user who wants do the same without learning a real programming language. [Aluigi Auriemma]) remember? that was some good words from you

3-Single_Raw
i read some post from some poeple, like (SIMS 4 STBL). if you adding a tag to disable write and do replace Raw_File(in this case, user cant import one file, that means user must extract all and import all, not just import some file[for update all size]. and you must do it in reimport2, not force mode. or maybe, should add a command-option to enable force mode in reimport2 mode, that would be awsome and its a big Process for quickbms! [just for text files and single files])

4-log file
i wish i could had a log file to read and keep my history of quickbms useage, its useul for some guys like me.

[aluigi]

1) unfortunately quickbms is not a magician and can't store 100 bytes in a place for 99 bytes. The only way to do that is the presence of an offset field that can be replaced with a new one pointing to the new data.

2) compressed and uncompressed sizes are fully supported in reimport2 when the Clog command is invoked, usually you have a compression flag in the archives that tells you if a file is compressed and I bet this is probably the case. It's up to you to understand the format and figure out what are the flag, compressed and uncompresse size fields.

Just for curiosity (not related to this case so ignore this phrase once avoiding confusion) there is only one rare case in which reimport2 can't work properly and it's when there is no compression flag, for example just a "if SIZE == ZSIZE ..." check but even in that case it's possible to make the script compatible by using comtype copy and clog instead of log.

3) honestly I have not understood but it looks something related to question 1 or you forgot to remove the non-edited files from the folder, quickbms reimports only the available files. While in extraction it's possible to filter the extracted files based on their names and extensions so if you have 1000000 files and only 10 jpg images you want to edit, it's better to use the -f "{}.jpg" option avoiding to extract 999990 useless files.

4) it depends by what type of log you need.
Launching quickbms with the -V option returns tons of useful technical information about the input data.
The -v option is rarely useful and it's related to the commands of the script.
They must be used from command-line maybe with the "less" cygwin tool (suggested) or with a simple redirect to file:

Code: Select all

quickbms.exe -V -0 script .bms input_file.dat > log.txt
quickbms.exe -V -0 script .bms input_file.dat | less

[Shokoniraya]

i totally get it, aluigi! thank you for explain!
and i just want to tell you: can you add force mode in reimport2?
because in reimport1 force, file size cant update...
half of problem solved in force mode, but if you add force mode in reimport2 too, i can solve that!

[aluigi]

reimport2 can't have a force mode because if the file is bigger it's automatically added at the end (without any questions to the user) and its offset/size fields rewritten.

I guess that what you want is a way for automatically replacing the offset/size fields and writing the file without caring what's the format, sort of "mix" of reimport+reimport2.
In that case it's better if you write a script for specifically doing that (not easy since it's not the job of quickbms) or just do it manually with a hex editor.

reimport.bat does not replace the original size field for compatibility with any format and for compatibility with the original behaviour of such feature: overwriting just the bytes of the file and nothing else.

[michalss]

i totally get it, aluigi! thank you for explain!
and i just want to tell you: can you add force mode in reimport2?
because in reimport1 force, file size cant update...
half of problem solved in force mode, but if you add force mode in reimport2 too, i can solve that!

I dont think you understand the reimport2 method. There is no such a force needed neither necessary as aluigi told you, when file is bigger its automatically go to end of the archive and in data table rewrite is size and offset, etc or other props needed. This method can't be used in same cases of segmented archives i guess..

[Shokoniraya]

I dont think you understand the reimport2 method. There is no such a force needed neither necessary as aluigi told you, when file is bigger its automatically go to end of the archive and in data table rewrite is size and offset, etc or other props needed. This method can't be used in same cases of segmented archives i guess..

dear michalss: reimport2 means; set new offset and size and i totaly know that
and force mode means replace raw file

but we cant do it on single files white reimport2!
need a command option to replace raw files like force mode, not write, just like text files! and just adding a tag for do this thing...

go to end of the archive and in data table rewrite is size and offset: and set old data as "00 00 00 00", i know that.

[aluigi]

What's probably not clear to many users is that the "force" option in reimport.bat is BAD and should be never used, NEVER.

It's something that corrupts the data and the only reason why it exists is that I preferred to give people a choice in case they really know what they are doing and that the edited archive will not work the 99.9% of times.
Please don't force me to remove that useless option from quickbms

Long story short: if you need to use the "force" option it means you are doing something completely wrong and you will receive no support.

That's the clear position about quickbms, but in your case, as already explained, you don't need quickbms to do something simple like that
Use a hex editor and do the job in one minute.

[aluigi]

I would like to introduce a better way to use the reimport and reimport2 modes by performing some additional checks.

1)
If you use reimport.bat for reimporting 1000 files and one of them is bigger than the original then the reimporting will be interrupted.
Possible solution:
- ask the user if he wants to retry (y/N/r/force) so he can edit the file and retry without running the process again

2)
If you have extracted 100000 files and have edited only one of them, quickbms will try to reimport all the 100000 files... it may be a problem for modders because deleting the non-edited files before reimporting may be a difficult (boring) process in some games.
Possible solution:
- calculate an hash (sha1 or sha256) on the archived file and an hash of the one on disk, compare them and reimport the file only if it's different. obviously if the size is different there is no need to make the hash.
While this solution seems nice, it has the disadvantage that basically quickbms needs to read (and decompress, just a full extraction without disk writing) the input file and compute the hash spending time and resources, multiply it by two since the same is done on the file on disk too.
sha1 is faster than sha256 and in this non-security context it's probably the best choice since no collisions are expected while editing a file.
I may implement this feature as an option for quickbms but I don't know if I have to add it to reimport.bat and reimport2.bat

[GHOST DEAD]

I dont think you understand the reimport2 method. There is no such a force needed neither necessary as aluigi told you, when file is bigger its automatically go to end of the archive and in data table rewrite is size and offset, etc or other props needed. This method can't be used in same cases of segmented archives i guess..

dear michalss: reimport2 means; set new offset and size and i totaly know that
and force mode means replace raw file

but we cant do it on single files white reimport2!
need a command option to replace raw files like force mode, not write, just like text files! and just adding a tag for do this thing...

go to end of the archive and in data table rewrite is size and offset: and set old data as "00 00 00 00", i know that.

yeah, right. but you can't do it with QuickBMS
Try to learn a language for making your own Tools

QuickBMS can't help in such thing: Text Files, Single Files, converting an File to another File Like PNG to tex. replace raw_data

[aluigi]

quickbms 0.9.1 will use the Capstone library as disassembler and will support every architecture supported by the library, so not just x86.

[aluigi]

reimport2 will work also with archives having the TOC at the end, maybe not 100% of times but most of them

[aluigi]

I still have no release date for quickbms 0.9.1 but at least I have finished to write the (temporary) changelog (the list of things fixed and added).
Even today I have fixed some bugs that have been casually found during a test.
I have also updated some libraries.

The only priority at the moment is beta testing for spotting new issues.
Hopefully next week I will have good news

[michalss]

I still have no release date for quickbms 0.9.1 but at least I have finished to write the (temporary) changelog (the list of things fixed and added).
Even today I have fixed some bugs that have been casually found during a test.
I have also updated some libraries.

The only priority at the moment is beta testing for spotting new issues.
Hopefully next week I will have good news

Are you also planning release fixed DDL with compress suport pls ? Waiting for this . thx

[aluigi]

Sure

[Shokoniraya]

yeah, right. but you can't do it with QuickBMS
Try to learn a language for making your own Tools

QuickBMS can't help in such thing: Text Files, Single Files, converting an File to another File Like PNG to tex. replace raw_data

i saw your posts. you had same problem, right? how did you solve it then? i cant do it

Sure

really? gonna be Great! even with text files? gonna possible?
cant wait to make new scripts with Good Stuff!