Shadow of Mordor *.Arch05 (x360)

[michalss]

Hi all,

Can anyone fix the aluigi script for SoM ? Thank you

Examples:

Code: Select all

https://dl.dropboxusercontent.com/u/38234344/discdata1.rar

[Echelo]

Hi all,

Can anyone fix the aluigi script for SoM ? Thank you

Examples:

Code: Select all

https://dl.dropboxusercontent.com/u/38234344/discdata1.rar

quickbms must use standard zlib as the default ComType, cause I don't see it being set in his script for the pc files.
Xbox360 is using ComType XMemDecompress, the files inside the .arch05 are big endian too. So types are like LDNB, not BNDL.

I don't think it would take too many modifications to get it to work for xbox360, aside from the fact the the xmem chunks have no clear decompressed size in the chunks, rather, their total decompressed and merged size is stored in the main table. Currently, in C# I'm using the main table size and subtracting 65536 from it as I process a chunk, then once the last chunk that is less than 65536 bytes is reached, I know I'm at the end.

Side note, PS3 version is using Deflate, and just adding ComType deflate to his current script and changing the endianness should work.

[Echelo]

Here is my unpacker so far. Only PS3 and Xbox360 support for now. It uses quickbms to handle the XMem chunks, so quickbms needs to be in the same folder as this exe.

This will unpack the .arch05 files to a folder named after them, and unpack any .embb inside them.
I've cleaned up the output to dispose of the temporary compressed files and only keep the decompressed final results. (Saves space on extraction).
Also the .embb files are now unpacked to layerX_unpacked_bundles, and are merged back to the original file structure the game engine uses.
I verified that all md5 match for overwritten files throughout all archives on both ps3 and xbox360. So now the tool doesn't write the file if it already exists. (Saves a little more time)

Code: Select all

usage: middleEarthArchiveUnpacker.exe <platform> <archive.arch05>
example: middleEarthArchiveUnpacker.exe PS3 layer01.arch05
example: middleEarthArchiveUnpacker.exe XBOX360 layer01.arch05

Now that the file structure is sorted, a repacker may be possible.
Its weird how this game has multiple copies of the same files throughout different bundles, (kind of wasteful it seems), but with that now known, I can make a repacker soon.

Attachment removed, see the repacker further down in this thread

[TON]

Here is my unpacker so far. Only PS3 and Xbox360 support for now. It uses quickbms to handle the XMem chunks, so quickbms needs to be in the same folder as this exe.

This will unpack the .arch05 files to a folder named after them, and unpack any .embb inside them to _d/_b. (was trying to keep unpacked paths short)
The files in the _d folders are the decompressed files from the .arch05, the files in the _b folders are the files from the .embb.

Usage:

Code: Select all

usage: middleEarthArchiveUnpacker.exe <platform> <archive.arch05>
example: middleEarthArchiveUnpacker.exe PS3 layer01.arch05
example: middleEarthArchiveUnpacker.exe XBOX360 layer01.arch05

Thanks Echelo!
Any chance for repacker?

[michalss]

yeah repack would be great

[aluigi]

Script updated to version 0.2.2, thanks.

[raykingnihong]

Script updated to version 0.2.2, thanks.

Hello, my friend, thank you for sharing

[michalss]

Script updated to version 0.2.2, thanks.

Lots of the things cannot be exported It only create header and files are always 1Kb of size, so it des not work correctly Also main arches does not work, it extract only 1 chunk 64kb

Examples :

Code: Select all

https://dl.dropboxusercontent.com/u/38234344/embb.rar

Archives:

Code: Select all

https://dl.dropboxusercontent.com/u/38234344/SoM_X360.rar

[michalss]

Ok i figuret it out. All files are chunked with different size... Here is template for 010....

Code: Select all

//--------------------------------------
//--- 010 Editor v3.1.1 Binary Template
//
// File:
// Author: michalss
// Revision:
// Purpose:
//--------------------------------------
local byte kompr;
local uint checker,trueer;

BigEndian();
    do {
        struct compfilestart {

            uint packed;
            trueer=FTell()+packed;
            if (trueer < FileSize()) {
                checker = ReadUShort(trueer);            
            } else {
                checker=0;    
            }
   
            if (checker==22616) {
                byte *SPAM*[packed+2];    
            } else {
                byte data[packed];
            }    


        } record;

    } while(FTell()!= FileSize());

[aluigi]

Well done michalss, indeed it's a chunked compression with 4 bytes alignment.
The alignment is probably per-file but, considering that all the stored files are aligned, I opted for a per-archive alignment which is exactly the same in this case.
Script 0.3
http://aluigi.org/papers/bms/others/sha ... mordor.bms

I removed also the creation of 0 bytes files.

[michalss]

Well done michalss, indeed it's a chunked compression with 4 bytes alignment.
The alignment is probably per-file but, considering that all the stored files are aligned, I opted for a per-archive alignment which is exactly the same in this case.
Script 0.3
http://aluigi.org/papers/bms/others/sha ... mordor.bms

I removed also the creation of 0 bytes files.

thank you, but still not there This chunk method is not normal. In some chunks in the end of the is XX number 22616. It is only on some of them and this breakpoint brake your extraction and decompression. If you look at my template you can see that im always checking if that breakpoint number is presen in the end of the chunk, coz if it is, then size of the chunk is about 2 bytes bigger. Not sure why this is like this, but ....

[aluigi]

I tried the script with the samples you provided and it worked, the "XX" is not a break but is just padding for the alignement.

[michalss]

ok try this file.... you will see https://dl.dropboxusercontent.com/u/382 ... r01.arch05

What you mean alligment pls?? Can you a bit explain pls ? How to calculated it?

[aluigi]

Some chunk sizes have AND 0xf0000000 instead of 0x80000000, script 0.3.1 out.

The padding/alignment is the amount of non-used bytes necessary to have data aligned.
If I have alignment 4 and I'm at offset 0x33 it means that the correct offset where reading data must be 0x34 so 1 byte unused.
If we are at offset 0 then it's 0, offset 1 is 4, offset 6 is 8 and so on.
The 'X' is the just unused byte and in fact you can see it only when the compressed data terminates in a non-aligned offset.

[aluigi]

In QuickBMS this is the math used for the calculation of alignment:

Code: Select all

if(var2 && ((u_int)var1 % (u_int)var2)) {
    var1 += (var2 - ((u_int)var1 % (u_int)var2));
}

where var1 is the offset and var2 is the alignment (for example 4).

[Echelo]

Here is my full unpacker and repacker. Have fun!

Code: Select all

usage: middleEarthArchiveTool.exe <mode> <archive.arch05> <platform>
example: middleEarthArchiveTool.exe unpack layer01.arch05 PS3
example: middleEarthArchiveTool.exe unpack layer01.arch05 XBOX360
example: middleEarthArchiveTool.exe repack layer01.arch05 PS3
example: middleEarthArchiveTool.exe repack layer01.arch05 XBOX360

This newer version will unpack all of the .arch05 files to a directory named after them.
Their internal .embb files will be unpacked to a "all_layers_unpacked_bundles" directory that is created when the first .embb is unpacked.
On repack, any files modded in the "all_layers_unpacked_bundles" are included in the repacked .embb before its packaged into the .arch05.

I've also tested with a few texture mods, and the game still loads the repacked .arch05.


At this time, it will repack XBOX360 .arch05 one to one if no files were modified.
For ps3, I think edge zlib 1.2.3.0 or just zlib1.dll version 1.2.3 is needed to make the repacked .arch05 one to one if no files were modded. Atm, I'm getting slightly better compression then the game. Will tinker with the zlib flavor later..

I can add PC support as well but I don't have a PC .arch05 to look at, I assume I can re-use my ps3 code and just change deflate to standard zlib with a few changes to endianness.

Remember to delete the install data and install it again with your repacked .arch05 files.

[aluigi]

You can find various samples here:
viewtopic.php?f=9&t=185

[michalss]

Here is my full unpacker and repacker. Have fun!

Code: Select all

usage: middleEarthArchiveTool.exe <mode> <archive.arch05> <platform>
example: middleEarthArchiveTool.exe unpack layer01.arch05 PS3
example: middleEarthArchiveTool.exe unpack layer01.arch05 XBOX360
example: middleEarthArchiveTool.exe repack layer01.arch05 PS3
example: middleEarthArchiveTool.exe repack layer01.arch05 XBOX360

This newer version will unpack all of the .arch05 files to a directory named after them.
Their internal .embb files will be unpacked to a "all_layers_unpacked_bundles" directory that is created when the first .embb is unpacked.
On repack, any files modded in the "all_layers_unpacked_bundles" are included in the repacked .embb before its packaged into the .arch05.

At this time, it will repack XBOX360 .arch05 one to one if no files were modified. I've also tested with a single texture mod and the game still loads the repacked .arch05.

For ps3, I think edge zlib 1.2.3.0 or just zlib1.dll version 1.2.3 is needed to make the repacked .arch05 one to one if no files were modded. Atm, I'm getting slightly better compression then the game. Will tinker with the zlib flavor later..

I can add PC support as well but I don't have a PC .arch05 to look at, I assume I can re-use my ps3 code and just change deflate to standard zlib with a few changes to endianness.

Remember to delete the install data and install it again with your repacked .arch05 files.

Thx mate gonna test it Soon All i need to repack correctly layer01 there is text and fonts ... Can i repack with with bigger size of text then original?

[Echelo]

Thx mate gonna test it Soon All i need to repack correctly layer01 there is text and fonts ... Can i repack with with bigger size of text then original?

Yes, the file tables for both the bundles and the archives are re-written on repack to reflect changes in file sizes and addresses.