Mass Effect Andromeda

[warrantyvoider]

this seems not to work for me, can you tell me an offset&size in a casfile, where this works for you?

I tried offset 0x3df791f, size 0x111df0 in mecdefaultinstallpackage\cas_01.cas and it works fine.
Maybe different bundles use different keys.

I can verify this works for me too, so yay, the keys are not install specific. also thank you again for the working code btw, you got pm

Yes, crypto key used is based on the "key id" in the catalog. The keys are hardcoded into the game exe, but you can find them in initfs_Win32, see Scripts/CasEncrypt.yaml.

hi rick, its been a while^^ thanks for the info!

How to clear data under VFS -> CHUNKS? "Load selected item to VFS" keeps on adding chunks every time it's used, and doesn't delete the old ones.

you cant yet, the problem is, theres no such thing as multiselect in treeview controls, so Ill code an extra dialog for that, but thats not my focus now, sry, just restart for now ¯\_(ツ)_/¯

[OClear]

yeah, from the 80 byte entries, the last 20 bytes are probably the encryption key. btw your zstd code doesnt work as it is, but if I put it into a file with ending .zst (starting with zstandard header) it can decompress it fine (f.e. zstd -d test.zst)

I have problems decompressing with zstd library. I get error "Unknown frame descriptor". I compile zstd library with legacy support and still get same error. My sample starts with magic id, so I don't know why it fails.

[warrantyvoider]

yeah, from the 80 byte entries, the last 20 bytes are probably the encryption key. btw your zstd code doesnt work as it is, but if I put it into a file with ending .zst (starting with zstandard header) it can decompress it fine (f.e. zstd -d test.zst)

I have problems decompressing with zstd library. I get error "Unknown frame descriptor". I compile zstd library with legacy support and still get same error. My sample starts with magic id, so I don't know why it fails.

this code assumes you give it compressed chunk data (with 0x0F70 header)

Code: Select all

using ZstdNet;
public static byte[] DecompressChunk(byte[] data)
        {
            MemoryStream m = new MemoryStream(data);
            MemoryStream res = new MemoryStream();
            while (m.Position < m.Length)
            {
                m.Seek(6, SeekOrigin.Current);
                ushort size = (ushort)(m.ReadByte() * 0x100 + m.ReadByte());
                byte[] buff = new byte[size];
                byte[] buff2 = null;
                m.Read(buff, 0, size);
                try
                {
                    using (var decompressor = new Decompressor())
                    {
                        buff2 = decompressor.Unwrap(buff);
                    }
                }
                catch { }
                if (buff2 != null)
                    res.Write(buff2, 0, buff2.Length);
            }
            return res.ToArray();
        }

https://github.com/skbkontur/ZstdNet

[warrantyvoider]

back home and back coding^^ started by making initFS browseable too



EDIT1:

Yay, I can load entire bundles in my VFS and display them! now I need to find one that uses encryption...



EDIT2:

now I can also decrypt the data on the fly and display it too, this is one of these "embargoed" items... (data\win32\loctext\en.toc -> .../tl_dlc -> load only res in bundle)
so now it automatically: decrypts, decompresses and puts all the pieces back together!



EDIT3: attached my current version; added an option to clear the VFS without restart; if you click on the top node in the structure view of a bundle and press F1 (shortcut for loading stuff), it will load the entire bundle into VFS. same goes for all ebx/res/chunk if you click a node named like that and single import if you pick one of its children nodes

PS: also thanks to everyone that contributed code and information so far, you will be added to the credits later, definitly!

[lyutor1945]

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

[Devisaur]

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

DAI Mod Tool is made by DawnlessSky who as said they don't plan on working on it.*

*Edited to fix initial mistake.

[lyutor1945]

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

DAI Mod Tool is made by WarrantyVoider who happens to be one spearheading, along with a few other people, the work in this forum.

Oh, sorry! Didn't know it. Looks like I'm goof:) Can I ask you another question then. WarrantyVoider and DawnlessSky it is one and the same person or different people?

[Devisaur]

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

DAI Mod Tool is made by WarrantyVoider who happens to be one spearheading, along with a few other people, the work in this forum.

Oh, sorry! Didn't know it. Looks like I'm goof:) Can I ask you another question then. WarrantyVoider and DawnlessSky it is one and the same person or different people?

I stand corrected I'm conflating two separate people -- but the answer to your original question can be found http://dawnatwork.tumblr.com/ --
"Hello!! I'm sure this questions has already been asked a hundred times, but will there be an update for DAI Tool or similar tool for Mass Effect Andromeda?
lyutor1945things

Not from me, no. Others will probably make one though."

[warrantyvoider]

simple answer, dawnless used my code and research, and made her own tool without mentioning me (afaik), so that tool came after DAItoolsWV (btw, the same goes for ehams tool, forgot how he called his)

greetz

[crackedmind]

back home and back coding^^ started by making initFS browseable too



EDIT1:

Yay, I can load entire bundles in my VFS and display them! now I need to find one that uses encryption...



EDIT2:

now I can also decrypt the data on the fly and display it too, this is one of these "embargoed" items... (data\win32\loctext\en.toc -> .../tl_dlc -> load only res in bundle)
so now it automatically: decrypts, decompresses and puts all the pieces back together!



EDIT3: attached my current version; added an option to clear the VFS without restart; if you click on the top node in the structure view of a bundle and press F1 (shortcut for loading stuff), it will load the entire bundle into VFS. same goes for all ebx/res/chunk if you click a node named like that and single import if you pick one of its children nodes

PS: also thanks to everyone that contributed code and information so far, you will be added to the credits later, definitly!

I'm writting my own tool with C++, using you algo. While debugging, i see, you are not correctly parsing .toc files. You suggest that types 0x82, 0x87 is completly different, but it's not true.

I found some information here. Type splits into flags and typecode. 0x82 - Flag = 0x4 Type=0x2
Flag 4 means that we need to skeep reading null terminated name for this field. Also, requiredChunks after that parses as list of 0x0F types (0x8F with flag).


Sample from my json dump

Code: Select all

        {
            "alwaysInstalled": false,
            "estimatedSize": 871620188,
            "friendlyName": 0,
            "id": "��\u0012Ɵ\u0019�%8��#�CTu",
            "installBundle": "Win32/streaminginstall/ayainstallpackage",
            "license": "",
            "mandatoryDLC": false,
            "maxSizeMB": -1,
            "name": "streaminginstall/ayainstallpackage",
            "optionalDLC": false,
            "requiredChunks": [
                "7�\t���\r��r��wڲ�",
                "?���\u0012C;�ڞ\\����\u0006",
                "GBWT+� �m\u0015���½L",
                "n��5؁����\"��\u001b�S",
                "n��5؁����;��\u001b�S",
                "n��5؁����7��\u001b�S",
                "q��(I�T���[㨍�[",
                "q��(I�T���[����[",
                "q��(I�T���[����[",
                "u����bu[\u001f�%���� ",
                "u����bu[\u001f�%���� ",
                "u����bu[\u001f�%���� ",
                "�\u0002��\u0012C���\u0012qX��d\u0010",
                "��f\u0001d���Q��",
                "��f\u0001d���Q��\u0001`�J\u0010",
                "��f\u0001d���Q��\u001fu�J\u0010",
                "ʗ��v6�8��}aFl�C",
                "��S�m?r\u001b����Z�2X",
                "��S�t?r\u001b����Z�2X",
                "��S�x?r\u001b����Z�2X"
            ],
            "saveLocked": true,
            "superbundles": [
                "Win32/game/levels/hubs/hub_aya_unc/hub_aya_unc_level",
                "Win32/game/levels/crit/crit_vlt/crit_vlt",
                "Win32/game/levels/hubs/hub_aya/hub_aya_level"
            ],
            "testDLC": false
        }

p.s. thanks to weiyun for decrypt code.

[OClear]

this code assumes you give it compressed chunk data (with 0x0F70 header)

Code: Select all

WORD w1;         // 0x100 or zero
WORD w2;
WORD comp_type;   // 0x700F
WORD w3;
// zstd magic id follows

-snip-
Solved.

[weiyun]

Does 0x100 mean there is an extra 256 bytes in the beginning? Dictionary?
Or maybe multi-part data? Or just unknown flags?

No, It's big endian, the actual value is 0x10000, which is the uncompressed chunk size.

[weiyun]

simple answer, dawnless used my code and research, and made her own tool without mentioning me (afaik), so that tool came after DAItoolsWV (btw, the same goes for ehams tool, forgot how he called his)

greetz

I ported your DAI code for stringtable and it works. Game uses TTF as font file this time.

[OClear]

Does 0x100 mean there is an extra 256 bytes in the beginning? Dictionary?
Or maybe multi-part data? Or just unknown flags?

No, It's big endian, the actual value is 0x10000, which is the uncompressed chunk size.

Ah, that makes sense. I didn't know what he was doing here:

Code: Select all

 ushort size = (ushort)(m.ReadByte() * 0x100 + m.ReadByte());

He could have read it as a big-endian word instead. It works out the same.

Is that the reason my decompressor is failing? Is the zstd compressed data written in big-endian format too?
CAT file LE, CAS file BE, game programmers can't write consistent data.

[Rick]

Is that the reason my decompressor is failing? Is the zstd compressed data written in big-endian format too?

Are you passing it too much/too little data?

Edit: to clarify, I've yet to have any issues with decompression of any data via Zstd.

https://github.com/gibbed/Gibbed.MassEffectAndromeda/blob/master/projects/Gibbed.Frostbite3.Unbundling/CompressionHelper.cs#L82

(note: my code is very much WIP)

[OClear]

Thanks, I didn't realize the zstd compressed data was split into multiple parts (failed when size >= 65536). I think I can get it working now.

[OClear]

Sample from my json dump

Do you have a tool/script that can convert .sb files into readable json format?

[crackedmind]

Do you have a tool/script that can convert .sb files into readable json format?

Not yet finished. I Planning to finish today.

[warrantyvoider]

I'm writting my own tool with C++, using you algo. While debugging, i see, you are not correctly parsing .toc files. You suggest that types 0x82, 0x87 is completly different, but it's not true.

I found some information here. Type splits into flags and typecode. 0x82 - Flag = 0x4 Type=0x2
Flag 4 means that we need to skeep reading null terminated name for this field. Also, requiredChunks after that parses as list of 0x0F types (0x8F with flag).


Sample from my json dump

Code: Select all

        {
            "alwaysInstalled": false,
            "estimatedSize": 871620188,
            "friendlyName": 0,
            "id": "��\u0012Ɵ\u0019�%8��#�CTu",
            "installBundle": "Win32/streaminginstall/ayainstallpackage",
            "license": "",
            "mandatoryDLC": false,
            "maxSizeMB": -1,
            "name": "streaminginstall/ayainstallpackage",
            "optionalDLC": false,
            "requiredChunks": [
                "7�\t���\r��r��wڲ�",
                "?���\u0012C;�ڞ\\����\u0006",
                "GBWT+� �m\u0015���½L",
                "n��5؁����\"��\u001b�S",
                "n��5؁����;��\u001b�S",
                "n��5؁����7��\u001b�S",
                "q��(I�T���[㨍�[",
                "q��(I�T���[����[",
                "q��(I�T���[����[",
                "u����bu[\u001f�%���� ",
                "u����bu[\u001f�%���� ",
                "u����bu[\u001f�%���� ",
                "�\u0002��\u0012C���\u0012qX��d\u0010",
                "��f\u0001d���Q��",
                "��f\u0001d���Q��\u0001`�J\u0010",
                "��f\u0001d���Q��\u001fu�J\u0010",
                "ʗ��v6�8��}aFl�C",
                "��S�m?r\u001b����Z�2X",
                "��S�t?r\u001b����Z�2X",
                "��S�x?r\u001b����Z�2X"
            ],
            "saveLocked": true,
            "superbundles": [
                "Win32/game/levels/hubs/hub_aya_unc/hub_aya_unc_level",
                "Win32/game/levels/crit/crit_vlt/crit_vlt",
                "Win32/game/levels/hubs/hub_aya/hub_aya_level"
            ],
            "testDLC": false
        }

you made me realize I have to rewrite that entire code for my tool, which im at currently, so thanks for the info


my reading class sofar, let me know if I do something wrong

Code: Select all

public static class FBJSON
    {
        public class Field
        {
            public byte flags;
            public byte type;
            public bool hasName;

            public string name;
            public ulong size;
            public object data;

            public Field(Stream s)
            {
                byte b = (byte)s.ReadByte();
                flags = (byte)(b >> 5);
                type = (byte)(b & 0x1f);
                hasName = (flags & 4) == 0;
                if (hasName) name = Helpers.ReadNullString(s);
                long start = s.Position;
                switch (type)
                {
                    case 1://list
                        size = Helpers.ReadLEB128(s);
                        data = new List<Field>();
                        while (s.Position - start < (long)size - 1)
                            ((List<Field>)data).Add(new Field(s));
                        s.ReadByte();
                        break;
                    case 2://dict
                        size = Helpers.ReadLEB128(s);
                        data = new Dictionary<string, Field>();
                        while (s.Position - start < (long)size - 1)
                        {
                            Field obj = new Field(s);
                            ((Dictionary<string, Field>)data).Add(obj.name, obj);
                        }
                        s.ReadByte();
                        break;
                    case 6://bool
                        data = s.ReadByte() == 1;
                        break;
                    case 7://binary string
                        size = Helpers.ReadLEB128(s);
                        data = Helpers.ReadNullString(s);
                        break;
                    case 8://int32
                        data = Helpers.ReadInt(s);
                        break;
                    case 5://unknown
                    case 9://int64
                        data = Helpers.ReadLong(s);
                        break;
                    case 0xf://uuid
                        data = new byte[0x10];
                        s.Read((byte[])data, 0, 0x10);
                        break;
                    case 0x10://sha1
                        data = new byte[0x14];
                        s.Read((byte[])data, 0, 0x14);
                        break;
                    case 0x13://payload blob
                        size = Helpers.ReadLEB128(s);
                        data = new byte[size];
                        s.Read((byte[])data, 0, (int)size);
                        break;
                }
            }


            public TreeNode[] ToNodes()
            {
                List<TreeNode> result = new List<TreeNode>();
                TreeNode t;
                int count = 0;
                switch (type)
                {
                    case 1:
                        foreach (Field f in (List<Field>)data)
                        {
                            t = new TreeNode((count++).ToString());
                            t.Nodes.AddRange(f.ToNodes());
                            result.Add(t);
                        }
                        break;
                    case 2:
                        foreach (KeyValuePair<string, Field> pair in ((Dictionary<string, Field>)data))
                        {
                            t = new TreeNode(pair.Key);
                            t.Nodes.AddRange(pair.Value.ToNodes());
                            result.Add(t);
                        }
                        break;
                    case 6:
                        result.Add(new TreeNode((bool)data ? "true" : "false"));
                        break;
                    case 7:
                        result.Add(new TreeNode((string)data));
                        break;
                    case 8:
                        result.Add(new TreeNode(((int)data).ToString("X8")));
                        break;
                    case 5:
                    case 9:
                        result.Add(new TreeNode(((long)data).ToString("X16")));
                        break;
                    case 0xf:
                    case 0x10:
                    case 0x13:
                        result.Add(new TreeNode(Helpers.ByteArrayToHexString((byte[])data)));
                        break;
                }
                return result.ToArray();
            }
        }

        public static List<Field> ReadFields(Stream s)
        {
            List<Field> result = new List<Field>();
            long len = s.Length;
            while (s.Position < len)
                    result.Add(new Field(s));
            return result;
        }
    }

EDIT: there are more types, stay tuned^^

[crackedmind]

you made me realize I have to rewrite that entire code for my tool, which im at currently, so thanks for the info

my reading class sofar, let me know if I do something wrong

Code: Select all

public static class FBJSON
    {
        public class Field
        {
            public byte flags;
            public byte type;
            public bool hasName;

            public string name;
            public ulong size;
            public object data;

            public Field(Stream s)
            {
                byte b = (byte)s.ReadByte();
                flags = (byte)(b >> 5);
                type = (byte)(b & 0x1f);
                hasName = (flags & 4) == 0;
                if (hasName) name = Helpers.ReadNullString(s);
                long start = s.Position;
                switch (type)
                {
                    case 1://list
                        size = Helpers.ReadLEB128(s);
                        data = new List<Field>();
                        while (s.Position - start < (long)size - 1)
                            ((List<Field>)data).Add(new Field(s));
                        s.ReadByte();
                        break;
                    case 2://dict
                        size = Helpers.ReadLEB128(s);
                        data = new Dictionary<string, Field>();
                        while (s.Position - start < (long)size - 1)
                        {
                            Field obj = new Field(s);
                            ((Dictionary<string, Field>)data).Add(obj.name, obj);
                        }
                        s.ReadByte();
                        break;
                    case 6://bool
                        data = s.ReadByte() == 1;
                        break;
                    case 7://binary string
                        size = Helpers.ReadLEB128(s);
                        data = Helpers.ReadNullString(s);
                        break;
                    case 8://int32
                        data = Helpers.ReadInt(s);
                        break;
                    case 5://unknown
                    case 9://int64
                        data = Helpers.ReadLong(s);
                        break;
                    case 0xf://uuid
                        data = new byte[0x10];
                        s.Read((byte[])data, 0, 0x10);
                        break;
                    case 0x10://sha1
                        data = new byte[0x14];
                        s.Read((byte[])data, 0, 0x14);
                        break;
                    case 0x13://payload blob
                        size = Helpers.ReadLEB128(s);
                        data = new byte[size];
                        s.Read((byte[])data, 0, (int)size);
                        break;
                }
            }


            public TreeNode[] ToNodes()
            {
                List<TreeNode> result = new List<TreeNode>();
                TreeNode t;
                int count = 0;
                switch (type)
                {
                    case 1:
                        foreach (Field f in (List<Field>)data)
                        {
                            t = new TreeNode((count++).ToString());
                            t.Nodes.AddRange(f.ToNodes());
                            result.Add(t);
                        }
                        break;
                    case 2:
                        foreach (KeyValuePair<string, Field> pair in ((Dictionary<string, Field>)data))
                        {
                            t = new TreeNode(pair.Key);
                            t.Nodes.AddRange(pair.Value.ToNodes());
                            result.Add(t);
                        }
                        break;
                    case 6:
                        result.Add(new TreeNode((bool)data ? "true" : "false"));
                        break;
                    case 7:
                        result.Add(new TreeNode((string)data));
                        break;
                    case 8:
                        result.Add(new TreeNode(((int)data).ToString("X8")));
                        break;
                    case 5:
                    case 9:
                        result.Add(new TreeNode(((long)data).ToString("X16")));
                        break;
                    case 0xf:
                    case 0x10:
                    case 0x13:
                        result.Add(new TreeNode(Helpers.ByteArrayToHexString((byte[])data)));
                        break;
                }
                return result.ToArray();
            }
        }

        public static List<Field> ReadFields(Stream s)
        {
            List<Field> result = new List<Field>();
            long len = s.Length;
            while (s.Position < len)
                    result.Add(new Field(s));
            return result;
        }
    }

Looks good and more simplier than before