Mass Effect Andromeda

Extraction and unpacking of game archives and compression, encryption, obfuscation, decoding of unknown files
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

warrantyvoider wrote:I had no time yet to look at the actual chunk contents, but as far as ive seen, each entry is missings its header and I dont see no magic for encryption anywhere. dunno, im busy with the two filesystems before I can look at content data

greetz

PS: about cas data:
Image

cat tells you to load a region from cas, then start reading blocks, with this 8 byte header. cat files can also contain a second list, where one entry is 80 bytes and has additional 20 bytes (in compare to a normal entry) that look like encryption keys again... (because no block header to see when reading where it says there, and also theres no cas number as far as I can see)



cat file 010 Editor Template

Code: Select all

uint flag; //0x01CED100
int zero;
byte signaturedata[0x224]; //rsa signature info, 0x224 bytes
byte nyan[0x10]; //NyanNyanNyanNyan, 0x10 bytes
int list1count;
int list3count;
int list2count;
byte padding[0xc];
struct {
byte sha1[0x14];
int offset;
int size;
int unknown;
int cas_num;
} list1[list1count];
struct {
byte sha1[0x14];
int offset;
int size;
int unknown1;
int unknown2;
int size2;  // size2 should equal to size
byte unk1[0x28];
} list2[list2count];
struct {
byte unknown[0x3c];
} list3[list3count];


cas is compressed in chunks using zstandard
format:

Code: Select all

int decompressedsize;
ushort flag?;
ushort compressedsize;
byte[] compresseddata; //compressed data starts with magic header 0xfd2fb528

Decompression code

Code: Select all

        [DllImport("libzstd.dll")]
        static extern UIntPtr ZSTD_decompress(IntPtr dst, int dstCapacity,
                              IntPtr src, int compressedSize);

        public static byte[] Decompress(byte[] buffer, long decompressedsize)
        {
            byte[] decbuffer = new byte[decompressedsize];
            int offset = 0;
            var cmphandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
            var decmphandle = GCHandle.Alloc(decbuffer, GCHandleType.Pinned);
            using (BinaryReaderEx br = new BinaryReaderEx(new MemoryStream(buffer), Endianness.Big))
            {
                while (br.BaseStream.Position < br.BaseStream.Length)
                {
                    int framedecsize = br.ReadInt32();
                    short flag = br.ReadInt16();
                    ushort framesize = br.ReadUInt16();
                    int decsize = (int)ZSTD_decompress(decmphandle.AddrOfPinnedObject()+ offset, decbuffer.Length,cmphandle.AddrOfPinnedObject() + (int)br.BaseStream.Position, (int)framesize);
                    offset += decsize;
                    br.BaseStream.Position += (int)framesize;
                }
            }
            cmphandle.Free();
            decmphandle.Free();
            return decbuffer;
        }
       


I decompressed the texture data with the correct size, but the pixel format seems wrong.

Image
warrantyvoider
Posts: 236
Joined: Tue Apr 04, 2017 11:44 am

Re: Mass Effect Andromeda

Post by warrantyvoider »

*SPAM*, really? zstandart? I tried so many others, damn^^ thank you, I will include this in my code, cat/cas structure was known, just not the algo

greetz

PS: have you seen a CAT with entries in the 3rd list?
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

warrantyvoider wrote:*SPAM*, really? zstandart? I tried so many others, damn^^ thank you, I will include this in my code, cat/cas structure was known, just not the algo

greetz

PS: have you seen a CAT with entries in the 3rd list?


Only cat in the patch directory has list3. It's related to resource patching.

Code: Select all

struct {
byte sha1_1[0x14];
byte sha1_original[0x14];
byte sha1_replacement[0x14];
} list3[list3count];

Where sha1_1 is unique. I think it serves as the id of this entry.
sha1_original is only found in original package.
sha1_replacement is found in list1 of the same package.
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

weiyun wrote:
warrantyvoider wrote:*SPAM*, really? zstandart? I tried so many others, damn^^ thank you, I will include this in my code, cat/cas structure was known, just not the algo

greetz

PS: have you seen a CAT with entries in the 3rd list?


Only cat in the patch directory has list3. It's related to resource patching.

Code: Select all

struct {
byte sha1_1[0x14];
byte sha1_original[0x14];
byte sha1_replacement[0x14];
} list3[list3count];

Where sha1_1 is unique. I think it serves as the id of this entry.
sha1_original is only found in original package.
sha1_replacement is found in list1 of the same package.


list1 of cat file has a lot of duplicated chunks, both within the same cat and across different cats.
list2 chunk seems encrypted.
OClear
Posts: 45
Joined: Tue Feb 16, 2016 6:36 pm

Re: Mass Effect Andromeda

Post by OClear »

weiyun wrote:

Code: Select all

struct {
byte unknown[0x3c];
} list3[list3count];



ayainstallpackage\cas.cat has (72999, 0, 31) entries. This third list has 80 bytes each, not 60.
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

OClear wrote:
weiyun wrote:

Code: Select all

struct {
byte unknown[0x3c];
} list3[list3count];



ayainstallpackage\cas.cat has (72999, 0, 31) entries. This third list has 80 bytes each, not 60.


Note that list3 count appears before list2count.
int list1count;
int list3count;
int list2count;
warrantyvoider
Posts: 236
Joined: Tue Apr 04, 2017 11:44 am

Re: Mass Effect Andromeda

Post by warrantyvoider »

yeah, from the 80 byte entries, the last 20 bytes are probably the encryption key. btw your zstd code doesnt work as it is, but if I put it into a file with ending .zst (starting with zstandard header) it can decompress it fine (f.e. zstd -d test.zst)
FartingSquirrel
Posts: 4
Joined: Fri Mar 18, 2016 11:02 am

Re: Mass Effect Andromeda

Post by FartingSquirrel »

Devisaur wrote:
FartingSquirrel wrote:Hey guys!
As I saw, you all were talking about the textures of the game, but is there anything about the text files for localization purposes? Did someone try to get them?


Not so far -- my main goal is to pull all the text files. I'd imagine what would need to occur is something similar to what they did for Dragon Age Inquisition(http://daitools.freeforums.org/). However it's a multiple step process to break it all down and I have yet to find a source that explains what they're doing/how they're doing it or even comments their code.

Well, it's good that someone's trying. Good luck with it.
Trip
Posts: 7
Joined: Thu Apr 06, 2017 12:50 pm

Re: Mass Effect Andromeda

Post by Trip »

Code: Select all

int decompressedsize;
ushort flag?;
ushort compressedsize;
byte[] compresseddata; //compressed data starts with magic header 0xfd2fb528


the flag is the compression format as noted by swinei on 1st page of this thread
warrantyvoider
Posts: 236
Joined: Tue Apr 04, 2017 11:44 am

Re: Mass Effect Andromeda

Post by warrantyvoider »

is there a list of which flag is what compression?
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

warrantyvoider wrote:is there a list of which flag is what compression?


I figured out the list2 is compressed using zstandard and then encrypted using AES 128 CBC Mode. That's why there is extra bytes added to the chunk. But I didn't look into key generation details yet.
warrantyvoider
Posts: 236
Joined: Tue Apr 04, 2017 11:44 am

Re: Mass Effect Andromeda

Post by warrantyvoider »

weiyun wrote:I figured out the list2 is compressed using zstandard and then encrypted using AES 128 CBC Mode. That's why there is extra bytes added to the chunk. But I didn't look into key generation details yet.


how did you figured out its AES 128 CBC? debugging? and what extra bytes do you mean? the ones at the end of the 80 byte entries in cat or the actual data in cas?

anyway, I added the compression algo to my program and can read all list1 entries fine, so all those chunks can be read and exported just fine now :D
Image

Flags from what ive seen sofar

Code: Select all

0x0F70 = ZStandard compressed
0x0071 = Uncompressed, not last block
0x0070 = Uncompressed, last block


greetz WV
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

warrantyvoider wrote:how did you figured out its AES 128 CBC? debugging? and what extra bytes do you mean? the ones at the end of the 80 byte entries in cat or the actual data in cas?


I debugged the exe file using x64dbg. It seems the key is fixed, but not hardcoded. I'm not sure if game uses other keys for decryption, but I didn't find any exception either.
The decrypted data is normal compressed chunk. The actual encrypted chunk data is padded to multiples of 16 since AES is a block cipher.

Decrypt code:

Code: Select all

public static byte[] Decrypt(Stream s, int size)
        {
            byte[] key = { 0x91, 0xC2, 0x1D, 0xC7, 0x65, 0x95, 0x0E, 0xB0, 0xC3, 0x38, 0xD7, 0x3D, 0xA6, 0xD7, 0x4B, 0x1C };
            var aes = AesCryptoServiceProvider.Create();
            aes.Mode = CipherMode.CBC;
            aes.KeySize = 128;
            using (CryptoStream cs = new CryptoStream(s, aes.CreateDecryptor(key, key), CryptoStreamMode.Read))
            {
                byte[] res = new byte[size];
                cs.Read(res, 0, size);
                return res;
            }

        }


It seems the conversation file has another layer of compression.
And the texture format code 0x42 doesn't look like DXT5.
OClear
Posts: 45
Joined: Tue Feb 16, 2016 6:36 pm

Re: Mass Effect Andromeda

Post by OClear »

This is just raw texture data from chunk with no header. Where is texture info?

256x256, DXGI_FORMAT_BC7_UNORM_SRGB (format 99)
Image

weiyun wrote:It seems the conversation file has another layer of compression.
And the texture format code 0x42 doesn't look like DXT5.


Where does "texture format code 0x42" come from?
Last edited by OClear on Thu Apr 06, 2017 11:47 pm, edited 4 times in total.
warrantyvoider
Posts: 236
Joined: Tue Apr 04, 2017 11:44 am

Re: Mass Effect Andromeda

Post by warrantyvoider »

weiyun wrote:I debugged the exe file using x64dbg. It seems the key is fixed, but not hardcoded. I'm not sure if game uses other keys for decryption, but I didn't find any exception either.
The decrypted data is normal compressed chunk. The actual encrypted chunk data is padded to multiples of 16 since AES is a block cipher.

Decrypt code:

Code: Select all

public static byte[] Decrypt(Stream s, int size)
        {
            byte[] key = { 0x91, 0xC2, 0x1D, 0xC7, 0x65, 0x95, 0x0E, 0xB0, 0xC3, 0x38, 0xD7, 0x3D, 0xA6, 0xD7, 0x4B, 0x1C };
            var aes = AesCryptoServiceProvider.Create();
            aes.Mode = CipherMode.CBC;
            aes.KeySize = 128;
            using (CryptoStream cs = new CryptoStream(s, aes.CreateDecryptor(key, key), CryptoStreamMode.Read))
            {
                byte[] res = new byte[size];
                cs.Read(res, 0, size);
                return res;
            }

        }


this seems not to work for me, can you tell me an offset&size in a casfile, where this works for you?

I tried:
offset:0x02EC3AC7
size:0x11F6C0
path:streaminginstall\ayainstallpackage\cas_02.cas

taken from the single 80 bytes entry at the end of the cat file in the same folder

I attached my cutout for compare
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

warrantyvoider wrote:this seems not to work for me, can you tell me an offset&size in a casfile, where this works for you?


I tried offset 0x3df791f, size 0x111df0 in mecdefaultinstallpackage\cas_01.cas and it works fine.
Maybe different bundles use different keys.
Rick
Posts: 8
Joined: Tue Aug 19, 2014 4:25 pm

Re: Mass Effect Andromeda

Post by Rick »

Yes, crypto key used is based on the "key id" in the catalog. The keys are hardcoded into the game exe, but you can find them in initfs_Win32, see Scripts/CasEncrypt.yaml.
OClear
Posts: 45
Joined: Tue Feb 16, 2016 6:36 pm

Re: Mass Effect Andromeda

Post by OClear »

warrantyvoider wrote:PS: attached my current version, I moved the CAT stuff from VFS (=Virtual File System= stuff in games memory) to FS (=FileSystem=real existing files) where it makes more sense


How to clear data under VFS -> CHUNKS? "Load selected item to VFS" keeps on adding chunks every time it's used, and doesn't delete the old ones.
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

Rick wrote:Yes, crypto key used is based on the "key id" in the catalog. The keys are hardcoded into the game exe, but you can find them in initfs_Win32, see Scripts/CasEncrypt.yaml.


Thanks for the info. Here is the yaml file. Can you find the key location in game exe? I didn't find any hardcoded keys.

Code: Select all

--- # encrypt embargoed assets (YAML format)
start:
 # Unique keys for each text language
 - key: 91c21dc765950eb0c338d73da6d74b1c
   keyid: d66a3e06
   filters:
     - game/localization/config/texttable/en/.*
 - key: ff67f81f63309f430756fa5ee8a49cf6
   keyid: 13f2f24d
   filters:
     - game/localization/config/texttable/br/.*
 - key: 09059e69d726ba9cc6cae5e7ea87959f
   keyid: e4b3b4dd
   filters:
     - game/localization/config/texttable/de/.*
 - key: 940fb798bcbc72375201ed143ab2338a
   keyid: 4608ff24
   filters:
     - game/localization/config/texttable/es/.*
 - key: 559813d02d2d498bda5c03161b82aba8
   keyid: 1e7a889c
   filters:
     - game/localization/config/texttable/fr/.*
 - key: 51724f6c718da5041214c3239c23a463
   keyid: 759bed13
   filters:
     - game/localization/config/texttable/it/.*
 - key: 216dad98a17d11e4b0175f1bc0828037
   keyid: 707deb43
   filters:
     - game/localization/config/texttable/pl/.*
 - key: ac7bb33878a14cc66daf80ff94b8089f
   keyid: 1d171a1b
   filters:
     - game/localization/config/texttable/ru/.*
 # Encrypt certain assets which are only available in the full game to prevent access to people playing the trial
 - key: 9d68ee1b349b8eb6ef4cf37197e1604d
   keyid: 827e55c8
   filters:
     # M2
     - game/levels/crit/crit_vlt/crit_vlt.*


Here is the revised template.

Code: Select all

//--------------------------------------
//--- 010 Editor v6.0.2 Binary Template
//
// File: Frostbite Cat
// Author: weiyun
// Revision:
// Purpose:
//--------------------------------------
uint flag; //0x01CED100
int zero;
byte signaturedata[0x224]; //rsa signature info, 0x224 bytes
byte nyan[0x10]; //NyanNyanNyanNyan, 0x10 bytes
int normalChunkCount;
int patchChunkCount;
int encryptedChunkCount;

byte padding[0xc];
struct {
byte sha1[0x14];
int offset;
int size;
int unknown;
int cas_num;
} normalChunk[normalChunkCount];
struct {
byte sha1[0x14];
int offset;
int size;
int zero;
byte casnum;
byte unknown[3];
int size2;  // size2 should equal to size
char keyid[8];
byte unk1[0x20];
} encryptedChunk[encryptedChunkCount];
struct {
byte sha1[0x14];
byte sha1_original[0x14];
byte sha1_patch[0x14];
} patchChunk[patchChunkCount];
weiyun
Posts: 19
Joined: Wed Aug 10, 2016 5:20 pm

Re: Mass Effect Andromeda

Post by weiyun »

OClear wrote:Where does "texture format code 0x42" come from?


It's in the itexture header chunk, its sha1 is located in the sb file.