Far Cry 3 Xbox360 LZMA decompression - Missing bytes at EOF?

[mlleemiles]

Hey, aluigi! I've been using your fat2_fat3 bms script for a while now and it's been working mostly fine! However it seems when decompressing certain files in .dat file, random numbers of bytes at EOF could go "missing", resulting incomplete extraction.

Here is an example for one file (generated\databases\generic\shopitemsubtype.lib):


(xbox360 vs pc and .lib file is cross platform in little endian)

Here is the archive I extracted it from: https://mega.nz/file/IXpygbzZ#XCKswR0YI ... FR6Gbamtfw

Another example when more bytes at EOF go missing (graphics\_materials\grassam-m-20120326110838.material.bin in multicommon.dat)

original_grassam-m-20120326110838.material.bin

I manually "fixed" it by comparing to PC file:

fixed_grassam-m-20120326110838.material.bin

Raw data stream cut from .dat file:
Filehash - 0x2b69ca0dbd7a1f5e;
Size - 1127;
ZSize - 490;
Offset - 99530512;

raw_grassam-m-20120326110838.material.bin

From what I understand, lzma_dynamic has to deal with truncated data but it isn't "perfect" as we see here. Is there a solution to this problem?

[mlleemiles]

Bump

[aluigi]

The error reported by the lzma decompressor is LZMA_STATUS_NEEDS_MORE_INPUT.
I remember that the farcry stuff was a mess and I can't say if the compressed stream was invalid (as I remember) or there was an additional byte that we need to read (I doubt, maybe try uploading a sample of 490+1 bytes).

lzma_dynamic is correct in its work based on the provided input data.

[mlleemiles]

The error reported by the lzma decompressor is LZMA_STATUS_NEEDS_MORE_INPUT.
I remember that the farcry stuff was a mess and I can't say if the compressed stream was invalid (as I remember) or there was an additional byte that we need to read (I doubt, maybe try uploading a sample of 490+1 bytes).

lzma_dynamic is correct in its work based on the provided input data.

Here is the raw data plus the next 5 bytes (basically the LZMA header of another compressed file). I found that when decompressing with lzma_dynamic, the last byte (of compressed data) isn't needed to get the same result, and the first byte of the raw data is the size (when it is not 0) of "uncompressed data" at EOF that usually needs to append to decompressed data.

However, there are many files starting with 0 and can't decompress correctly

[mlleemiles]

Here is an example of file with "uncompressed data"

Filehash - 0x46aa7a5dbd72be90;
Size - 1126;
ZSize - 484;
Offset - 99558983;