Problem with reimport using witcher3.bms

[tsanford01]

Hello,
First off Very Great job! on this toolset. The tool extracts everything perfectly, however the reimport fails do to file size. I tried it by just extracting down to the file I wanted to edit and tried to reimport WITHOUT any edits just export then import and still get a file size difference error. This is what I get on importing the unchanged file. Any help would be greatly appreciated.

E:\Quickbms and Script-14-1-0>quickbms -G -w -r
QuickBMS generic files extractor and reimporter 0.6.3a
by Luigi Auriemma
e-mail: me@aluigi.org
web: aluigi.org
(May 6 2015 - 10:55:55)

http://quickbms.aluigi.org
http://twitter.com/luigi_auriemma
http://zenhax.com

- GUI mode activated, remember that the tool works also from command-line
where are available various options like folder scanning, filters and so on
- select the BMS script or plugin to use
- select the input archives/files to extract, type * or "" for whole folder and
subfolders
- select the output folder where extracting the files
- REIMPORT mode enabled!
- open input file E:\Quickbms and Script-14-1-0\patch.bundle
- open script E:\Quickbms and Script-14-1-0\witcher3.bms
- set output folder E:\Quickbms and Script-14-1-0\00_mod

offset filesize filename
--------------------------------------
Error: file "gameplay\items\def_loot_containers.xml"
the reimport option acts as a reimporter and so you cannot reinsert a
file if it's bigger than the original otherwise it will overwrite the
rest of the archive or cannot be loaded correctly:
new size: 31442 (1179468 uncompressed)
old size: 27897 (1179468 uncompressed)
- do you want to skip this file? (y/N/force)
y will continue with the next file
N (default) will terminate QuickBMS
force will force the reimporting of the file

Thanks,
Tom

[Ekey]

You can't reimport if modified file size is larger than the original.

new size: 31442 (1179468 uncompressed)
old size: 27897 (1179468 uncompressed)

[tsanford01]

I know that the problem is the file was not modified or even opened just extracted then reimported and the filesize came up different.

[aluigi]

There is no solution for that problem.
QuickBMS already uses the best compression algorithms available, in this case lz4 and doboz are just the original algorithms and zlib is just a mix of 3 algorithms of which the best one is used.

I don't know what algorithm is used for that specific file, let me know.

[tsanford01]

Thank you for your response. Quickbms seems to work fine on all the smaller files in the same .bundle they are all xmls and work on both export and import. it is the larger ones that seem to cause the problems. Very strange, the .bundle wouldn't have different compressions in the same archive I assume.

[aluigi]

Can you provide a link to that patch.bundle or any other archive that gives the same problem in reimport mode?
I really doubt to be able to do something but, who knows, maybe there is an option to use in doboz or lz4 to compress better and I didn't use it... just guessing.

[tsanford01]

Thanks again for taking your time to help with this if possible. I have included a file xml.bundle that contains the file def_loot_containers.xml which is the exact file as before but the bundle itself is smaller. Just FYI I have successfully imported several of the smaller files in that bundle it is the larger ones that pose a challenge.

Included xml.bundle,witcher3.bms used, and extracted file in question.

https://bitbucket.org/tsanford01/witcher-projects/downloads/Witcher%20Import%20problem.rar

Best regards,
Tom

[aluigi]

Thanks a lot.
Unfortunately I don't have good news.

The file has a size of 1178586 bytes and we must shrink it to 27910 bytes.

The following are the results obtained by calling the functions of lz4:

Code: Select all

LZ4HC  31493
LZ4HC2 30619
LZ4    52918

The first thing to notice is that lz4hc2 compresses more than the lz4hc used in quickbms.
I don't know if I missed this function or wasn't available when I implemented it in quickbms, but it will be used for sure in the next version.

As you can see we are still very far from the 27910 we need to obtain.

[aluigi]

I have an update.
I have just used the latest lz4 (the one in quickbms is just few weeks old) and I have better results:

Code: Select all

LZ4HC  27910
LZ4HC2 27813
LZ4    52918

This is an excellent news, the problem is solved

I guess I will release the new quickbms quite soon so you can start the modding.

[tsanford01]

You are awesome !! Thanks for all your help. I look forward to using your excellent tool more soon

Regards,
Tom

[aluigi]

I have a bad new about Witcher 3.
The following is what I wrote to another user to explain why the reimported files don't work:

Code: Select all

the problem is just the lz4 algorithm so it's not something caused by
quickbms.

If you compress a file with lz4, add some random data at its end and
then try to decompress it, you willl get the same error.

I guess that lz4 doesn't have a end marker like zlib and so it
decompresses everything it gets till it reaches an error like the end
of the output buffer.

As already said, I can do nothing about it because Witcher uses the
same library and in fact the reimported file doesn't work as you tested.
The reason that it doesn't work even if you didn't modify it is caused
by the compression used in quickbms that is better than the one of the
game (same algorithm with better compression option) and so the
compressed file is smaller than the original but the compressed size
is the same of the original file resulting in "garbage" after the new
file.

The only solution is using a rebuilder, I have already seen some solutions on github.

[tsanford01]

Thanks again for your hard work !.. I have been working with this and noticed the problem as well. I have been working with a couple of bundles that contain .xml documents. It works fine on smaller files but not the larger ones get the extra garbage data. Looking at the hex lz4 seems to add a header or key of some sort to the beginning and several bytes to the end. Using the lz4 library right off of github I can compress the file then hex edit it to remove extra bits and it works fine. Very strange issue however.

Best regards,
Tom

P.s. - I have to hex edit out the garbage created after the file and replace all bits with 00 to make up the correct file size.

[tsanford01]

Hello again sorry to drag this on. But, I had a thought. We are trying to use the latest LZ4 for compression to get the correct size and the release just before the latest wouldn't work. The Witcher 3 has been in development for years, so my question is how are we certain LZ4 was used and the very latest release at that? As you can tell I am a hobby programmer no pro by any standard, so I am not questioning your skill or anything just trying to learn something. As for end marker and such, the bundle holds a file path into the bundle, an offset, a compressed size and a decompressed size, so I figure it just reads the amount of data between offset and end of compressed size.


Regards,

Tom

[aluigi]

Witcher 3 uses LZ4_compress_HC probably with compressionLevel 9, if you use it you should notice that the size of the compressed file is the same of the original.

I guess you are writing a rebuilder right?
You should have no problems in that case, the problem exists only with reinjectors.

[tsanford01]

Hello again,

Yes I a attempting to rebuild some files by relocating them in the bundle changing offset, size, etc. What I am finding is that minor changes to a file say change a single variable from 1 to 9 etc. Then recompile with LZ4 r129 using HC at 9 I still get a file with many extra bytes of data even after removing the header key etc. I just can't get them to work in game with a single change. Even removing comments and such to make them the exact same size and leaving where them are suppose to be using both with your nice Quickbms import or direct hex edit it just won't work, very frustrating. These are simple xml files too.. Using hex workshop and comparing files that single change causes several differences in the file. And ideas?

I of course am not asking you to help mod the game. I am just not understanding the LZ4 compression I guess. Llooking at the posted frame format:

A compliant decompressor must be able to decompress at least one working set of parameters that conforms to the specifications presented here. It may also ignore checksums. Whenever it does not support a specific parameter within the compressed stream, it must produce a non-ambiguous error code and associated error message explaining which parameter is unsupported

https://github.com/Cyan4973/lz4/blob/master/lz4_Frame_format.md
So, I am wondering how your tool is able to decompress the files without parameters besides that it is an LZ4 compression. Is the magic number, frame descriptor, etc. not needed? That would give me a clue as to how to recompress it properly. Like possible a header located in another file.. just a guess.

P.s. - Any file I have found compressed with zlib can be edited very easily. I can move them, expand them, whatever.

regards,
Tom

[aluigi]

quickbms simply uses this code to decompress:

Code: Select all

size = LZ4_decompress_safe(in, out, zsize, size);

Remember that you don't have to use the same compression algorithm and you don't even have to use compression at all, after you have edited offset/zsize/size edit also the flag.

[tsanford01]

Simply a brilliant idea.. lose the troublesome LZ4 altogether change the flags go zlib like the easy to manipulate files and roll on.

You have been great help!
Thanks,
Tom