REIMPORT2 mode - no size limits

[aluigi]

doesn't exist "force" in reimport2.
NEVER use "force" in reimport mode.

[happydance]

wow, this is really great news!

cant wait to try it out.

[aluigi]

quickbms 0.8.4 will return a warning if you use reimport2 on sequential content, for example chunked files or files saved with "savepos OFFSET ; log output.dat OFFSET 0x100" and so on. That will avoid to corrupt the archive without giving any explanation to the user since currently (0.8.3) the new data is appended at the end of the file and the game will still read the old content due to the lack of an offset reference (it's sequential).

Additionally it will also zero the original space if the file is bigger, that will save space when you will distribute the compressed edited archive.
I wasn't sure if enabling or not this feature but it offers also the advantage of knowing if something is wrong with the reimport operation, let's say something doesn't work as expected and the game tries to read the old content, it will be just zeroes (invalid) and it's easy to notice it (crash, errors, corrupted images/text).

Hopefully there will be no bugs with these two changes

[RC-3197]

This is an amazing step forward, now if only we could add new files then it would be perfect!

[aluigi]

That's impossible because you can only edit the existent information.
Adding new files require the full 100% knowledge of the specific target format (and version) and a whole rebuilder.

[unominhtuan]

I used the reimport with a sequential file .... after the file was translated even though the new file = the original file size (uncompress size) .... but I was told "compress size bigger". If I make the smaller volume, my reimport file gets inserted "null" characters at the end and the game can not read ... do i have any solution for this?

[aluigi]

reimport2 can't be used for files stored in sequential way.
You need to use reimport.bat and being sure that your compressed/uncompressed files is smaller/equal than the original file.
Remember to use the backup copy if you already tried reimport/reimport2 and it failed, just to be sure it didn't corrupt the file.

[unominhtuan]

reimport2 can't be used for files stored in sequential way.
You need to use reimport.bat and being sure that your compressed/uncompressed files is smaller/equal than the original file.
Remember to use the backup copy if you already tried reimport/reimport2 and it failed, just to be sure it didn't corrupt the file.

i use reimport.bat The size of the text file is smaller than the original file, file gets inserted "null" characters at the end, and the game can not read this file... only text file. model, font, video...do not have this problem

[aluigi]

That's exactly how it must work so it's perfect from my side.
Your file is short and therefore it fills the difference (old size - new size) with zeroes.
If you want to avoid that make a file with the same size of the original putting maybe spaces at the end if it's a text file.

[unominhtuan]

Thanks aluigi, I'm also using this way

[unominhtuan]

That's exactly how it must work so it's perfect from my side.
Your file is short and therefore it fills the difference (old size - new size) with zeroes.
If you want to avoid that make a file with the same size of the original putting maybe spaces at the end if it's a text file.

sorry for bothering me many times, i really have a problem. After "add spaces" to new uncompress size = old uncompress file, but only a number of files active ... some files are still quickbms automatically added "null" ... my game can not load the file containing the signature self "null" ....
I add spaces with hex editor, and make sure the new file = the original file
You can modify my quickbms.exe file. Instead of adding "null", it will automatically add "spaces" if the size is smaller than the original file. I wait for help from you
This is the image depicting the problem

[aluigi]

If quickbms adds null bytes it means that the size of the file is not the same of the original one. Simple.

Do you have script and archive for doing a test? Upload them.
I can't promise nothing because if quickbms does that it means it's perfectly correct, I just want to do a check in case there is a rare bug.

[unominhtuan]

If quickbms adds null bytes it means that the size of the file is not the same of the original one. Simple.

Do you have script and archive for doing a test? Upload them.
I can't promise nothing because if quickbms does that it means it's perfectly correct, I just want to do a check in case there is a rare bug.

i use this script http://aluigi.zenhax.com/bms/qq_sfc.bms

I have made the uncompress size = uncompress size of the original file. But, "compress size" is not equal, compress size is smaller than original file can reimport .... I have no way to both "uncompress size" and "compress size" = original file ... I use hex editor only Can edit uncompress size = original file

Or you can check this script so that it can get names, folders (not sequentially). Then I can freely reimport the file using reimport2.bat

[aluigi]

You can't use the reimport feature with that script for various reasons:
- sequential files require reimport.bat (no compressed/uncompressed size field modification, so no reimport2.bat)
- lz4 requires that the specified compressed/uncompressed size perfectly match the available data

Obviously it's rare that the compressed size matches the original one since it depends by the entropy of the file and the various settings and implementations of the compression algorithm.
Long story short: there is no solution except for manual hex editing or just writing a rebuilder from scratch.

[unominhtuan]

You can't use the reimport feature with that script for various reasons:
- sequential files require reimport.bat (no compressed/uncompressed size field modification, so no reimport2.bat)
- lz4 requires that the specified compressed/uncompressed size perfectly match the available data

Obviously it's rare that the compressed size matches the original one since it depends by the entropy of the file and the various settings and implementations of the compression algorithm.
Long story short: there is no solution except for manual hex editing or just writing a rebuilder from scratch.

I still have to do you again ...
After using the hex editor, edit the size = the original file by "add spaces"
this is the original file:


file editing:


file after reimport:


  I think there was a bug in the script or quickbms that caused quickbms to auto-fix, adding "null" to make it bigger than the original file. from 16,777 to 16,813 bytes

Here are 3 files: original, my edit, after reimport:
https://drive.google.com/open?id=1HuEBcsYq-OBQC8vswJhNN5GPDgrOo3Gn

You can check it again
thanks

[aluigi]

Uhmmm no, there is no bug.

You can use only reimport.bat, NOT reimport2.bat as I said.

You didn't provide the original archive but I made a simple test here because the format is very simple so I created an archive from scratch and everything (extract->reimport->reextract) worked perfectly as expected.

If you have that wrong result with reimport.bat and quickbms 0.9.0 then upload the original archive.

[unominhtuan]

Uhmmm no, there is no bug.

You can use only reimport.bat, NOT reimport2.bat as I said.

You didn't provide the original archive but I made a simple test here because the format is very simple so I created an archive from scratch and everything (extract->reimport->reextract) worked perfectly as expected.

If you have that wrong result with reimport.bat and quickbms 0.9.0 then upload the original archive.

i use reimport.bat and quickbms 0.90, not reimport 2.
Why reextract files (after reimport.dev) are resized?
original archive https://drive.google.com/open?id=1JAwJ7wKo7O_JA8W85YOUJYKviwBzVXtW

[aluigi]

Finally I can replicate the issue with the archive and I confirm what I wrote few posts above:

- lz4 requires that the specified compressed/uncompressed size perfectly match the available data

Now the technical explanation:

quickbms uses the following code for performing the decompression:

Code: Select all

size = LZ4_decompress_safe_partial(in, out, zsize, size, *outsize);

where size is the expected decompressed size (for example 0x4189 in your case) while *outsize is the size of the output buffer that can be any number bigger than size (for example 0x41ad).

lz4 doesn't have a termination bit so it continues to extract data till the result is:

Code: Select all

targetOutputSize >= result < dstCapacity

Code: Select all

/*!
LZ4_decompress_safe_partial() :
    This function decompress a compressed block of size 'srcSize' at position 'src'
    into destination buffer 'dst' of size 'dstCapacity'.
    The function will decompress a minimum of 'targetOutputSize' bytes, and stop after that.
    However, it's not accurate, and may write more than 'targetOutputSize' (but always <= dstCapacity).
   @return : the number of bytes decoded in the destination buffer (necessarily <= dstCapacity)
        Note : this number can also be < targetOutputSize, if compressed block contains less data.
            Therefore, always control how many bytes were decoded.
            If source stream is detected malformed, function returns a negative result.
            This function is protected against malicious data packets.
*/
LZ4LIB_API int LZ4_decompress_safe_partial (const char* src, char* dst, int srcSize, int targetOutputSize, int dstCapacity);

Your reimported file DOES NOT have the matching compressed size 0xa97, it has something smaller like 0xa8d but still lz4 will decompress bytes which means also all the padded data between 0xa97 and 0xa8d till the dstCapacity is full.
That's why you see the zeroes in your reextracted file.

quickbms is perfectly correct in what it does, indeed your game isn't able to load the edited file due to the unmatching compressed size.

End of the technical explanation.

[unominhtuan]

Finally I can replicate the issue with the archive and I confirm what I wrote few posts above:

- lz4 requires that the specified compressed/uncompressed size perfectly match the available data

Now the technical explanation:

quickbms uses the following code for performing the decompression:

Code: Select all

size = LZ4_decompress_safe_partial(in, out, zsize, size, *outsize);

where size is the expected decompressed size (for example 0x4189 in your case) while *outsize is the size of the output buffer that can be any number bigger than size (for example 0x41ad).

lz4 doesn't have a termination bit so it continues to extract data till the result is:

Code: Select all

targetOutputSize >= result < dstCapacity

Code: Select all

/*!
LZ4_decompress_safe_partial() :
    This function decompress a compressed block of size 'srcSize' at position 'src'
    into destination buffer 'dst' of size 'dstCapacity'.
    The function will decompress a minimum of 'targetOutputSize' bytes, and stop after that.
    However, it's not accurate, and may write more than 'targetOutputSize' (but always <= dstCapacity).
   @return : the number of bytes decoded in the destination buffer (necessarily <= dstCapacity)
        Note : this number can also be < targetOutputSize, if compressed block contains less data.
            Therefore, always control how many bytes were decoded.
            If source stream is detected malformed, function returns a negative result.
            This function is protected against malicious data packets.
*/
LZ4LIB_API int LZ4_decompress_safe_partial (const char* src, char* dst, int srcSize, int targetOutputSize, int dstCapacity);

Your reimported file DOES NOT have the matching compressed size 0xa97, it has something smaller like 0xa8d but still lz4 will decompress bytes which means also all the padded data between 0xa97 and 0xa8d till the dstCapacity is full.
That's why you see the zeroes in your reextracted file.

quickbms is perfectly correct in what it does, indeed your game isn't able to load the edited file due to the unmatching compressed size.

End of the technical explanation.

I understand, sad that there is no solution. thank you

[Qnoops]

Im getting this error with unreal script, do you have any solution?