The Division 2 SDF File format

[lunger]

Hello, I have been digging into the archives for The Division 2 and have hit a wall. It appears that the compressed TOC chunk in the sdf.sdftoc file is no longer a simple zlib decompress. The beginning of the chunk does not have the 0x78 0xDA signature anymore. The compression ratio of the chunk seems to be better, closer to 2.27 versus 2.10 of Div1 chunk. The first 32bits of the chunk are 0xA01713FF, which doesn't seem to match any magic numbers from other compression algorithms. I see in the script, that LZ4 is being used in other snowdrop engine games with FileVersions of >= 23. The FileVersion for Div1's sdftoc file is 22. The Div2 sdftoc file is 37.

It would make sense that they switched to LZ4 compression in order to improve load times, but the magic number isn't there, plus I would expect the compression ratio to go down versus increase nearly 10%. Only thing I can think of is that the compressed code has been encrypted or obfuscated. :/

[lunger]

Here are sample files to play with. I tried a bunch of different things. Thinking maybe it was zstd compressed as magicless but that resulted in a failed decompress... not sure what else to try.

sdf.sdftoc
https://www114.zippyshare.com/v/RMMN6OD0/file.html
sdf-A-0000.sdfdata
https://www114.zippyshare.com/v/Kg1cw22Z/file.html

[aluigi]

What's the offset of the chunk you referred?

I see a lot of zlib data there.

[lunger]

I have it calculated the same with both your script and a go utility I started writing to replace the rouge_sdf extractor, 438262. Version 22 appeared to have the 0x78 0xDA signature in the middle of the larger chunk as well.

Code: Select all

go run main.go extract -i '/d/Games/Ubisoft Game/hunter/sdf/pc/data' -o /e/divisiondata/div2_beta
Input directory: /d/Games/Ubisoft Game/hunter/sdf/pc/data
Output directory: /e/divisiondata/div2_beta

++++++++++++++++++++++++++++++++++++++++++++++++ Begin extraction ++++++++++++++++++++++++++++++++++++++++++++++++
2019/02/06 12:44:49 toc.go:90: FileInfo: name=sdf.sdftoc size=42316037
2019/02/06 12:44:49 toc.go:114: Header: {FileTag:1414743383 FileVersion:37 DecompressedSize:94950714 CompressedSize:41877727 Zero:0 BlockOneCount:2940 DdsHeaderBlockCount:1718}
2019/02/06 12:44:49 toc.go:129: Header2: {NewKey1:31 NewKey2:1921939 NewKey3:133091 NewKey4:1245186 NewKey5:2949151}
2019/02/06 12:44:49 toc.go:144: Id: {Massive:28559167928361325 Data:[219 221 93 56 19 0 0 0 155 40 65 69 235 102 210 205 159 124 138 46 116 148 84 12 223 137 150 83 140 242 134 193] Ubisoft:32763726162125429}
2019/02/06 12:44:49 toc.go:145: CompressedSize: 41877727
2019/02/06 12:44:49 toc.go:146: DecompressedSize: 94950714
2019/02/06 12:44:49 toc.go:147: CompressionRatio: 2.267332083233648
2019/02/06 12:44:49 toc.go:150: Current Position: signature flag 96
2019/02/06 12:44:49 toc.go:165: Current Position: signature 97
2019/02/06 12:44:49 toc.go:167: sign flag: 1
2019/02/06 12:44:49 toc.go:172: Current Position: ddsheaders 177126
2019/02/06 12:44:49 toc.go:186: ddsHeaderBlock length: 0
2019/02/06 12:44:49 toc.go:191: Current Position: compressed content 438262
2019/02/06 12:44:49 toc.go:195: toc.readContent failed zlib: invalid header

Code: Select all

QuickBMS generic files extractor and reimporter 0.9.2
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Nov 25 2018 - 11:53:16)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                     @zenhax @quickbms  Twitter & Scripts

- start the scanning of the input folder: D:\Games\Ubisoft Game\hunter\sdf\pc\data
- open input file D:\Games\Ubisoft Game\hunter\sdf\pc\data\.\sdf-A-0000.sdfdata
- open script .\ubisoft_sdf2.bms

  offset   filesize   filename
--------------------------------------
- enter in folder D:\Games\Ubisoft Game\hunter\sdf\pc\data
  coverage file 0     0%   0          469215143  . offset 00000000
- open input file D:\Games\Ubisoft Game\hunter\sdf\pc\data\sdf.sdftoc
VERSION: 37
UNCOMPRESSED_SIZE: 94950714
COMPRESSED_SIZE: 41877727
OFFSET: 438262
comptype: zlib

Error: the compressed zlib/deflate input is wrong or incomplete (-3)
Info:  algorithm   1
       offset      0006aff6
       input size  0x027f00df 41877727
       output size 0x05a8d53a 94950714
       result      0xffffffff -1

Error: the uncompressed data (-1) is bigger than the allocated buffer (94950714)

Last script line before the error or that produced the error:
  41  clog MEMORY_FILE OFFSET INFO_ZSIZE INFO_SIZE
  

[aluigi]

I guess that offset is wrong.

Then I tought you were referring to sdf-A-0000.sdfdata while in that example it's sdf.sdftoc.
As far as I can see, the compressed or encrypted data in sdf.sdftoc starts from 0x27c8e.

[lunger]

Oh, I screwed up, I sent you the offset for the latest toc file, not the sample I gave you... the offset that I was using for the sample I gave you is 427510 / 0x685F6. I have too many sdftoc files on my machine...

So you think that might be the start of the toc chunk? Where the first 32bits are:

Code: Select all

B8 9C 31 1A

The Compressed Size of the file matches up with the FileSize - the ID tag Size - CompressedSize as with the other files. It aligns perfectly after the DDS header block. Which is the offset I listed above.

Code: Select all

++++++++++++++++++++++++++++++++++++++++++++++++ Begin extraction ++++++++++++++++++++++++++++++++++++++++++++++++
2019/02/06 12:44:25 toc.go:90: FileInfo: name=sdf.sdftoc size=28115072
2019/02/06 12:44:25 toc.go:114: Header: {FileTag:1414743383 FileVersion:37 DecompressedSize:59330859 CompressedSize:27687514 Zero:0 BlockOneCount:2966 DdsHeaderBlockCount:1637}
2019/02/06 12:44:25 toc.go:129: Header2: {NewKey1:31 NewKey2:1723695 NewKey3:788450 NewKey4:1441805 NewKey5:1245197}
2019/02/06 12:44:25 toc.go:144: Id: {Massive:28559167928361325 Data:[144 151 28 5 21 0 0 0 13 212 150 112 151 5 47 38 106 1 195 222 153 50 184 168 247 66 73 171 27 55 151 142] Ubisoft:32763726162125429}
2019/02/06 12:44:25 toc.go:145: CompressedSize: 27687514
2019/02/06 12:44:25 toc.go:146: DecompressedSize: 59330859
2019/02/06 12:44:25 toc.go:147: CompressionRatio: 2.1428741850929627
2019/02/06 12:44:25 toc.go:150: Current Position: signature flag 96
2019/02/06 12:44:25 toc.go:165: Current Position: signature 97
2019/02/06 12:44:25 toc.go:167: sign flag: 1
2019/02/06 12:44:25 toc.go:172: Current Position: ddsheaders 178686
2019/02/06 12:44:25 toc.go:186: ddsHeaderBlock length: 0
2019/02/06 12:44:25 toc.go:191: Current Position: compressed content 427510
2019/02/06 12:44:25 toc.go:195: toc.readContent failed zlib: invalid header

[lunger]

One thing that is interesting is these files have an additional 20 bytes between the Header of TD1 and the ID byte string which I am assuming are 5 new 32bit integers that are part of the file header. Haven't been able to determine what they are for though.

[seb851]

Any news ?

[lunger]

Here is all the debug information I could collect about the structure of the sdftoc file from the sample above.

Code: Select all

++++++++++++++++++++++++++++++++++++++++++++++++ Begin extraction ++++++++++++++++++++++++++++++++++++++++++++++++
2019/02/11 17:45:44 toc.go:104: FileInfo: name=sdf.sdftoc size=28115072
2019/02/11 17:45:44 toc.go:128: Header: {FileTag:1414743383 FileVersion:37 DecompressedSize:59330859 CompressedSize:27687514 Zero:0 BlockOneCount:2966 DdsHeaderBlockCount:1637}
2019/02/11 17:45:44 toc.go:378: OFFSET [header2]: 28 / 0X1C
2019/02/11 17:45:44 toc.go:145: Header2: {NewKey1:31 NewKey2:1723695 NewKey3:788450 NewKey4:1441805 NewKey5:1245197}
2019/02/11 17:45:44 toc.go:378: OFFSET [id]: 48 / 0X30
2019/02/11 17:45:44 toc.go:162: Id: {Massive:28559167928361325 Data:[144 151 28 5 21 0 0 0 13 212 150 112 151 5 47 38 106 1 195 222 153 50 184 168 247 66 73 171 27 55 151 142] Ubisoft:32763726162125429}
2019/02/11 17:45:44 toc.go:163: CompressedSize: 27687514
2019/02/11 17:45:44 toc.go:164: DecompressedSize: 59330859
2019/02/11 17:45:44 toc.go:165: CompressionRatio: 2.1428741850929627
2019/02/11 17:45:44 toc.go:378: OFFSET [sig_exists]: 96 / 0X60
2019/02/11 17:45:44 toc.go:378: OFFSET [sig]: 97 / 0X61
2019/02/11 17:35:55 toc.go:201: Signature: {SignatureExists:1 Signature:[]
2019/02/11 17:45:44 toc.go:378: OFFSET [block1]: 726 / 0X2D6
2019/02/11 17:45:44 toc.go:378: OFFSET [idBlock]: 12590 / 0X312E
2019/02/11 17:45:44 toc.go:378: OFFSET [block1_2]: 154958 / 0X25D4E
2019/02/11 17:45:44 toc.go:378: OFFSET [dds_headers]: 178686 / 0X2B9FE
2019/02/11 17:45:44 toc.go:378: OFFSET [toc_data]: 427510 / 0X685F6
2019/02/11 17:45:44 toc.go:237: toc.readContent failed zlib: invalid header

The same debug output for the sdftoc of the private beta:

Code: Select all

++++++++++++++++++++++++++++++++++++++++++++++++ Begin extraction ++++++++++++++++++++++++++++++++++++++++++++++++
2019/02/11 17:35:55 toc.go:104: FileInfo: name=sdf.sdftoc size=42316037
2019/02/11 17:35:55 toc.go:128: Header: {FileTag:1414743383 FileVersion:37 DecompressedSize:94950714 CompressedSize:41877727 Zero:0 BlockOneCount:2940 DdsHeaderBlockCount:1718}
2019/02/11 17:35:55 toc.go:378: OFFSET [header2]: 28 / 0X1C
2019/02/11 17:35:55 toc.go:145: Header2: {NewKey1:31 NewKey2:1921939 NewKey3:133091 NewKey4:1245186 NewKey5:2949151}
2019/02/11 17:35:55 toc.go:378: OFFSET [id]: 48 / 0X30
2019/02/11 17:35:55 toc.go:162: Id: {Massive:28559167928361325 Data:[219 221 93 56 19 0 0 0 155 40 65 69 235 102 210 205 159 124 138 46 116 148 84 12 223 137 150 83 140 242 134 193] Ubisoft:32763726162125429}
2019/02/11 17:35:55 toc.go:163: CompressedSize: 41877727
2019/02/11 17:35:55 toc.go:164: DecompressedSize: 94950714
2019/02/11 17:35:55 toc.go:165: CompressionRatio: 2.267332083233648
2019/02/11 17:35:55 toc.go:378: OFFSET [sig_exists]: 96 / 0X60
2019/02/11 17:35:55 toc.go:378: OFFSET [sig]: 97 / 0X61
2019/02/11 17:35:55 toc.go:201: Signature: {SignatureExists:1 Signature:[]
2019/02/11 17:35:55 toc.go:378: OFFSET [block1]: 726 / 0X2D6
2019/02/11 17:35:55 toc.go:378: OFFSET [idBlock]: 12486 / 0X30C6
2019/02/11 17:35:55 toc.go:378: OFFSET [block1_2]: 153606 / 0X25806
2019/02/11 17:35:55 toc.go:378: OFFSET [dds_headers]: 177126 / 0X2B3E6
2019/02/11 17:35:55 toc.go:378: OFFSET [toc_data]: 438262 / 0X6AFF6
2019/02/11 17:35:55 toc.go:237: toc.readContent failed zlib: invalid header

Also, there is a new feature of the engine that supports live patches. There is a config.dat file found within `My Documents` from the live patch they completed during the private beta:

Code: Select all

{
    "hash": "emDhBMmjbg0rOsKj5d558+HgZhM9bBarbW/cOmOMEwHUIGHyTV+GG9I4qzBiziegJIn/doHsv9Icx/VmQno7VTDXlArRFPE68cR95OOB6HC7apbekU+a9WaRVEgXNCp2A35ZFbYXg3/k4Ka4JG9uHlkjse9GI//KCy+MxEdLlQR80De9h4Ft+6bE8fjuqlD0wZojpHyozAob/p5CrBqxvh8wCc1umwsC2JyrQg2MWWOAER9W7ihaRlRuFt++s+lXD4+429DV4/qyPeNM5Um0/UYQAHIyuK1HyjnQCNxgzGc5v9PPFl6DF8Y6dz0Dve/zE3EkKUxtFNJEFEA/nAwfHA==",
    "config": {
        "files": [
            {
                "size": "20875",
                "name": "sdf-A-0005.sdfdata",
                "slices": [
                    {
                        "size": "20875",
                        "hash": "3ccdf3c38ac1b08f0e9ad2688c8cef8d79d5e7ada173df404d3539ceb485a4d4",
                        "path": "27773156-4CD5-4141-BC3BAAFD44F706D5_0_0"
                    }
                ]
            },
            {
                "size": "255847",
                "name": "sdf.sdftoc",
                "slices": [
                    {
                        "size": "255847",
                        "hash": "2fb98ed7df7e0a1fc0338657e2589ecd142148fce0dae83f4d5f95d6f742e966",
                        "path": "27773156-4CD5-4141-BC3BAAFD44F706D5_1_0"
                    }
                ]
            }
        ],
        "name": "untitled",
        "sdf_guid": {
            "sizeshash": "14831029683549710491",
            "totalsize": "82550054363",
            "filenameshash": "888498253216775327",
            "archivemagic": "13945099980711037407"
        },
        "id": "27773156-4CD5-4141-BC3BAAFD44F706D5",
        "transfer_type": "3",
        "base_url": "http://pc-content.prod.tctd2.ubisoft.com/secure/prod/pc/27773156-4CD5-4141-BC3BAAFD44F706D5/"
    }
}

The sdftoc file in this directory has the standard 0x78DA zlib header which is interesting.

I attempted to use the `archivemagic` key from the config file as a XOR mask for the first 64bits of the compressed chunk but it did not prove fruitful for the sample I provided above or for the private beta files.

[seb851]

any update ?

[txt]

The format didnt change alot, but theres still some stuff i dont know. Heres some info:

the toc37 header has some new additions:
- BuildVersion - 32bit (game's build)
- BuildDate - 16bit year, 16bit month, 16bit day
- BuildTime - 16bit hour, 16bit minute, 16bit seconds


After the HasSignature bool there seems to be a IsEncrypted bool, then comes the signature as normal.

After the signature theres a 0x134 part of data that i just skip over as of now. At the end of it there seems to be a list of languages in the main toc.

After the dds headers theres a new array with datacount/block0 as length, which is 8 bytes per element. Havent looked at this closely, looks a bit like 2 32bit integers


Then comes the main compressed data.
If the isencrypted bool is 0, then you can decompress it normally, if not then you have to decrypt the buffer before decompressing.
I havent been able to extract the keys as of now...

[seb851]

Any update ? i can wait to datamine !!

[Yannerrins]

Game is available to pre-install. I hope files will be openable right now...

[end360]

Here's the files posted earlier but from the pre-loaded game files:

sdf.sdftoc
https://mega.nz/#!uFVxBQDZ!eE_kT1cWCQOq ... vGZDV3ZUg8
sdf-A-0000.sdfdata
https://mega.nz/#!HENXlKbI!VMVS4nEF2ynj ... wEefXyx7DM

[Sir Kane]

For legal reasons, this does not include the AES key and IV. When you get then, put them inside sd_key.txt and sd_iv.txt respectively.

Please also note that you'll need to get a copy of oo2core_7_win64.dll (renamed older/newer versions might also work).

Download

[seb851]

For legal reasons, this does not include the AES key and IV. When you get then, put them inside sd_key.txt and sd_iv.txt respectively.

Please also note that you'll need to get a copy of oo2core_7_win64.dll (renamed older/newer versions might also work).

Download

Nice Troll

[Sir Kane]

What?

[Invoker]

What?

He's upset because everyones refused to give him access due to him posting random information from the files to reddit

[blenux]

Any updates to this?, wanting to look at some of the models in 3ds max or Blender etc.

[theranger2]

For legal reasons, this does not include the AES key and IV. When you get then, put them inside sd_key.txt and sd_iv.txt respectively.

Please also note that you'll need to get a copy of oo2core_7_win64.dll (renamed older/newer versions might also work).

Download

Can anyone please share how to get the AES key? Or nobody knows how to do that?