Renowned Explorers .tim archive

[zespri]

As you can see Table_Compressed_lzhamtest.dat it differs from the original only in the existing header with 13 bytes.

Well, I personally can't see that. These two files in the archive are very different. There are indeed 26 bytes in common after the first extra 13, but after that I cannot see any resemblance. The size is also quite different.

[abbeybas]

My apologies for leaving this thread for so long.

So indeed we use 10.8.1. We did not modify the original algorithm.

To decompress we use the following flags:

Code: Select all

decompressParams.m_struct_size = sizeof(lzham_decompress_params);
decompressParams.m_dict_size_log2 = dictionarySize;
decompressParams.m_table_update_rate = LZHAM_FASTEST_TABLE_UPDATE_RATE;
decompressParams.m_decompress_flags = 0;
decompressParams.m_num_seed_bytes = 0;
decompressParams.m_pSeed_bytes = nullptr;
decompressParams.m_table_max_update_interval = 0;
decompressParams.m_table_update_interval_slow_rate = 0;

We use variable sized compression dictionaries. In all places where compression is used the dictionarySize is calculated based on the size of the input with:

Code: Select all

std::min<uint32_t>(std::max<uint32_t>(floor_to_pow_2(uncompressedSize), LZHAM_MIN_DICT_SIZE_LOG2), 25);

floor_to_pow_2 is defined as:

Code: Select all

// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLogLookup
uint32_t floor_to_pow_2(uint32_t v)
{
  static const char LogTable256[256] =
  {
#define LT(n) n, n, n, n, n, n, n, n, n, n, n, n, n, n, n, n
    - 1, 0, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
    LT(4), LT(5), LT(5), LT(6), LT(6), LT(6), LT(6),
    LT(7), LT(7), LT(7), LT(7), LT(7), LT(7), LT(7), LT(7)
#undef LT
  };

  unsigned int t, tt; // temporaries

  if ((tt = (v >> 16)) != 0)
    return (t = (tt >> 8)) ? 24 + LogTable256[t] : 16 + LogTable256[tt];

  return (t = v >> 8) ? 8 + LogTable256[t] : LogTable256[v];
}

I hope this helps you guys!

[aluigi]

Thanks abbeybas, that explains why only the first bytes where correctly extracted.
It's the first time I used lzham and I guess that field can be easily brute forced to guess the correct value for the extraction.
I will try. thanks

[zespri]

Just tried that, it works. Brute forcing is not likely to work as sometimes you do not get with wrong dictionary size, just gibberish. It could work if we always knew the exact uncompressed size but it looks that it is not always known. Thanks.

[aluigi]

lzham is really painful.
From lzham.h line 144:

// IMPORTANT: The values of m_dict_size_log2, m_table_update_rate, m_table_max_update_interval, and m_table_update_interval_slow_rate MUST
// match during compression and decompression. The codec does not verify these values for you, if you don't use the same settings during
// decompression it will fail (usually with a LZHAM_DECOMP_STATUS_FAILED_BAD_CODE error).

Now, regarding this particular situation with this game, I have implemented the brute forcer but I'm still unable to get the correct decompression because the result is never LZHAM_DECOMP_STATUS_SUCCESS (3 or 4). The max I get is LZHAM_DECOMP_STATUS_NOT_FINISHED.

I'm doing my tests with the Table_Compressed_Original.dat file provided by Ekey.
Using the parameters 19 and 20 I get the correct 0x8000 bytes but not the rest of the file which is 876726 bytes.

[aluigi]

Update: got the exact result.
I had to use lzham_decompress_memory instead of lzham_decompress

[zespri]

Now, regarding this particular situation with this game, I have implemented the brute forcer but I'm still unable to get the correct decompression because the result is never LZHAM_DECOMP_STATUS_SUCCESS (3 or 4). The max I get is LZHAM_DECOMP_STATUS_NOT_FINISHED.

You need to make sure that you set

Code: Select all

decompressParams.m_table_update_rate = LZHAM_FASTEST_TABLE_UPDATE_RATE;

This is NOT the default.

[aluigi]

yes, the "20" was referred to that value. I forgot to say that 19 was the dictionary size and 20 the table update rate.

I have tested also the brute forcer and works perfectly

[aluigi]

I have the full working script that I have just tested on the provided content.tim:
http://aluigi.org/papers/bms/others/abbeygames.bms

Obviously you have to wait the next version of quickbms that will be probably just later today, in case there are other last-minute things to fix on the fly.

[aluigi]

QuickBMS 0.6.7a is now available. Feel free to use it with the script.

[Ekey]

[0wn3df1x]

I have the full working script that I have just tested on the provided content.tim:
http://aluigi.org/papers/bms/others/abbeygames.bms

Obviously you have to wait the next version of quickbms that will be probably just later today, in case there are other last-minute things to fix on the fly.

Thank you!

But there is a problem with lua files.
Without these files, the archive is successfully packed and work into the game.

With lua files archive is packed too, but don't work into the game.
Possible unpacking error? Or packing?

[aluigi]

There are no problems with the extraction, I checked the extracted lua files and they are ok.

Do you refer to the reimporting feature?
In that case there is a problem, yes, because lzham by default uses a brute forcer and you have to use the same settings expected by the game (check the previous post of the developers) to perform a correct reimporting.
I opted for the brute forcing because it's the faster way but are you sure the game doesn't work with only the extracted files?

[happyend]

hi~aluigi~script not support win10?

[aluigi]

provide more info like screenshots and details of the error.

[abbeybas]

By extracting the contents of the .tim file into the root of the game folder you can run (and modify) the game with the extracted files. There is no need to repack to modify the game.

[0wn3df1x]

By extracting the contents of the .tim file into the root of the game folder you can run (and modify) the game with the extracted files. There is no need to repack to modify the game.

I del tim file and >>
Try to put files into Renowned Explorers\data - Crash.
Try to put into Renowned Explorers\ - Crash.

[zespri]

By extracting the contents of the .tim file into the root of the game folder you can run (and modify) the game with the extracted files. There is no need to repack to modify the game.

I del tim file and >>
Try to put files into Renowned Explorers\data - Crash.
Try to put into Renowned Explorers\ - Crash.

I dunno, works for me. Need to put it into Renowned Explorers\ not to Renowned Explorers\data
The game requires content.tim file, so it won't work without it, but you can put a minimal one instead of the full one (attached). And you do not need at all if you want to replace just a few files - it just works.

[zespri]

So with Bas's awesome help (who explained me that the hash table in the file uses Murmur Hash) I was able to pack the files back into a functioning archive. Not a quickbms script, sorry, a C# one. I published it here: https://github.com/AndrewSav/ReisUnpack

[0wn3df1x]

So with Bas's awesome help (who explained me that the hash table in the file uses Murmur Hash) I was able to pack the files back into a functioning archive. Not a quickbms script, sorry, a C# one. I published it here: https://github.com/AndrewSav/ReisUnpack

Thanks!

Bad news: I can't repack .tim.
Good news: With C# unpacker game work with the extracted files.