Ubisoft file format (FAT2/FAT3)

[aluigi]

Ubisoft use a specific file format for their games, you can recognize it by the 2TAF and 3TAF magic signature at the beginning of the archive.

The format is known to be used by:
FarCry 3 - FAT2
The Crew - FAT2
Watch Dogs - FAT3

The format is divided in two files:
- FAT containing the index table
- DAT containing only the data referenced by FAT

One of the bad things of formats used by multiple games and multiple platforms is the existence of multiple customizations.
For example the same archive on Xbox360, PS3 and PC is handled in different ways.

This difference is related mainly to the handling of compressed files because FAT3 on Xbox uses LZX (aka xmem), PC too (it's rare, maybe they use the same DX sdk) and PS uses zlib.
To make the situation even more difficult is the fact that the compression is chunked so there is no compressed and uncompressed size but a sequence of compressed parts of the file.

FAT2 is easier because doesn't use chunks but also in that case there are at least 2 different compression algorithms: lzo and lzma.

A bad, very bad, thing of this format is that its files are nameless, there is only a name CRC to identify them.

There are various tools to handle the format for specific games, moreover due to the name CRC limitation.

I worked on FAT2/3 just recently and I release a script for quickbms that should be compatible with all the games and all or almost all the platforms:
http://aluigi.org/papers/bms/fat2_fat3.bms

For reference, the following is a FAT2 file from The Crew:

Code: Select all

32 54 41 46 05 00 00 00 01 00 00 00 2a 10 00 00   2TAF........*...
61 0d 00 00 00 00 00 00 10 d1 b9 82 73 fa 06 00   a...........s...
16 02 00 c0 3c 70 62 00 b1 e3 01 00 00 00 00 00   ....<pb.........
30 6c d8 3c 14 af 15 00 73 4e 00 40 a7 a5 90 00   0l.<....sN.@....
61 3a 00 00 00 00 00 00 b5 dc b4 62 73 27 1d 00   a:.........bs'..
1d 08 00 c0 d7 a3 24 00 e1 b4 00 00 00 00 00 00   ......$.........
cb b5 da 79 88 ec 2d 00 cc 16 00 c0 f4 a7 57 00   ...y..-.......W.
91 f1 00 00 00 00 00 00 c2 a6 dc 01 56 e1 37 00   ............V.7.
8b 21 00 40 24 f6 79 00 81 ad 0a 00 00 00 00 00   .!.@$.y.........
76 a5 e1 df a6 dd 48 00 f9 64 02 40 c4 be 0b 00   v.....H..d.@....

While the following is a FAT3 from Watch Dogs:

Code: Select all

33 54 41 46 08 00 00 00 03 04 37 00 ab 12 00 00   3TAF......7.....
3b 06 23 00 82 05 02 00 00 11 00 80 1a 9d 97 00   ;.#.............
5c 02 30 00 82 05 02 00 f0 11 00 80 c2 54 97 00   \.0..........T..
3f 99 3e 00 82 85 00 00 30 03 00 e0 4b 24 17 00   ?.>.....0...K$..
9b 7d 52 00 00 00 00 00 a0 01 00 00 7d 03 20 00   .}R.........}. .
28 0e 63 00 82 05 02 00 30 11 00 80 20 0c 97 00   (.c.....0... ...
fa f3 67 00 82 05 08 00 c0 46 00 80 2d 23 83 00   ..g......F..-#..
95 04 6a 00 82 05 10 00 a0 89 00 80 33 5d 82 00   ..j.........3]..
cf 5f 6a 00 82 05 10 00 a0 cf 00 80 23 fa 8b 00   ._j.........#...
7b a3 6e 00 0a 1a 04 00 11 34 00 60 4c 0a 1f 00   {.n......4.`L...

[cenkzenk]

Hi How to open .loc file

[CobraGamer]

Could you take a look at the format on The Crew 2 archives? This BMS script doesn't work with them.
I'd be really thankful if you could release an updated script for this!

[aluigi]

oodle compression, script 0.1.5:
http://aluigi.org/bms/fat2_fat3.bms

[CobraGamer]

Thank you!

[cyberspeed]

... A bad, very bad, thing of this format is that its files are nameless, there is only a name CRC to identify them.
...

Would it be a very bad idea if the script could extract them for their names as they are now but with their CRC64Bit hashes?
Would help a lot to identify them would be very helpful if you could update the script to support that.

[aluigi]

Sure, at line 244 replace 'set NAME string ""' with:

Code: Select all

xmath TMP_CRC "NAME_CRC >> 32"
string NAME p "%08x%08x" TMP_CRC NAME_CRC

Additionally I'm quite sure there is a bug at line 75 where "get DUMMY longlong" should be "get NAME_CRC longlong" other version 9 wouldn't have the correct crc associated.
Please let me know how it works.

[cyberspeed]

Sure, at line 244 replace 'set NAME string ""' with:....
...a bug at line 75 where "get DUMMY longlong" should be "get NAME_CRC longlong" ...
Please let me know how it works.

I would like to confirm both recommendations above work as expected, and the results are accurate. Thank you so much. The script can definitely be updated for future use that way no one will get confused what the files are as i did
Before:

Code: Select all

  Offset           Filesize   Name
  00000000170f8ea7 263333     0000000000000000.dat
  000000003e7acfa3 22         0000000000000001.dat
  00000000247ba630 485613     0000000000000002.dat
  0000000083451d28 134246     0000000000000003.dat
  00000000836031e7 263333     0000000000000004.dat

After:

Code: Select all

  Offest           FileSize   CRC64
  00000000170f8ea7 263333     0002CE399C7095C2
  000000003e7acfa3 22         00035ACDB463C178
  00000000247ba630 485613     0005161C58F1FAE8
  0000000083451d28 134246     000A94756572D236
  00000000836031e7 263333     000E9033917CE2FF

All these Ubisoft games built on Dunia engine, have actual file names but need to be manually edited/added, so how can i feed the list with the names so the assets can be extracted with file/folder structure based on their reversed hashes?

Example hashes above would be extracted like so in their file/folder structure:

Code: Select all

project\road66\base\area\area_01\area_01_06\secteur_01_06_06_00\detailid_01_06_06_00.xbt
project\road66\generated\area\area_02\area_02_10\secteur_02_10_05_01\ai_02_10_05_01.van
project\road66\generated\area\area_01\area_01_09\secteur_01_09_07_01\ai_01_09_07_01.van
project\road66\generated\area\area_05\area_05_11\secteur_05_11_02_03\displacement_05_11_02_03.hms
project\road66\base\area\area_05\area_05_11\secteur_05_11_02_05\detailid_05_11_02_05.xbt

I tried something like:

Code: Select all

quickbms.exe script.bms filenames.txt input output

And few more variations, but to no avail, is there a way to do it? I cant figure it out

[aluigi]

Perfect.

For that feature you mention exists the NameCRC command.
I don't know what crc/hash algorithm is used on these filenames and if it's supported by the crc engine of quickbms.
If you know the algorithm filenames.txt must contain only the filenames, otherwise it must contain a hexadecimal crc followed by the name (like "1122334455667788 path\file.txt").

In that case would be enough to replace the whole content of the GET_NAME_FROM_CRC function with:
namecrc NAME NAME_CRC "filenames.txt" 64

[cyberspeed]

Perfect....I don't know what crc/hash algorithm is used on these filenames and if it's supported by the crc engine of quickbms....

Based on: http://aluigi.altervista.org/bms/quickb ... engine.txt looks like its supported, its the Jones variant

In that case would be enough to replace the whole content of the GET_NAME_FROM_CRC function with:
namecrc NAME NAME_CRC "filenames.txt" 64

I did try, yet every single assets gets overwritten by a name like: (2).bin
And not sure i fully understand how to modify the function
Te names inside the filenames.txt are with out any other characters besides the actual path as described above for those 5 CRC64 hashes, no ""

[aluigi]

So I guess this is the correct command:
namecrc NAME NAME_CRC "filenames.txt" 64 0xad93d23594c935a9 "64 0 0 0 1 0"

Regarding the function I meant, from:

Code: Select all

startfunction GET_NAME_FROM_CRC
    set NAME string ""
    if NFO_EXISTS != 0
        for z = 0 < NAMES
            getarray TMP_CRC 10 z
            if TMP_CRC == NAME_CRC
                getarray NAME 11 z
                math z = NAMES  # break is bugged
            endif
        next z
    endif
endfunction

To:

Code: Select all

startfunction GET_NAME_FROM_CRC
    namecrc NAME NAME_CRC "filenames.txt" 64 0xad93d23594c935a9 "64 0 0 0 1 0"
    if NAME == ""
        xmath TMP_CRC "NAME_CRC >> 32"
        string NAME p "%08x%08x" TMP_CRC NAME_CRC
    endif
endfunction

[cyberspeed]

Thank you, i tested both TC games and it works flawlessly.

[cyberspeed]

The script will NOT export strings that happen to have a space in the entire full path, it gets skipped, how can i avoid that?
Ideal would be to take in to consideration entire string line, spaces or not.
Can that be achieved?

[cyberspeed]

Would anyone else happen to know how to make the script extract the strings that have spaces in name at all?
Right now the whole string is skipped.

[spiritovod]

I suppose more explanation about the issue is required. I think cyberspeed is talking about situation, when list with filenames contains strings with spaces, but CRC is not calculated properly for them. In the same time, explicit declaration works fine. For example, for CRC32 with default parameters:
[extracted\test file.bin] with NAME_CRC = 0x402C6A32 -> doesn't work (output will be 402C6A32.dat)
[0x402C6A32 extracted\test file.bin] with NAME_CRC = 0x402C6A32 -> works fine (output will be "extracted\test file.bin")

@cyberspeed: If you can calculate those crc manually, you can use list with pre-defined crc values, this way it will work as expected.

[cyberspeed]

...I think cyberspeed is talking about situation, when list with filenames contains strings with spaces, but CRC is not calculated properly for them...

That is correct.

@cyberspeed: If you can calculate those crc manually, you can use list with pre-defined crc values, this way it will work as expected.

I always assumed the script is written to not support spaces, or that it would require more code to support them, using your suggestion works as it should, i just had no idea it would require the HASH to be included even though quickbms natively supports that CRC64 algo. Thanks non the less.
I was confused when Luigi said above, if the CRC is unknown the CRC64 hash must be provided but if its supported by quickbms it is not necessary, so now i know.

[spiritovod]

@cyberspeed: Um, no, it's more like a quickbms bug. Like explained in the documentation, you may use list with paths only and list with pre-defined values + paths, both should work in the same way for supported crc schemes. For example:
[extracted\test2.bin] with NAME_CRC= 0x2AB02897 -> works fine (output will be "extracted\test2.bin")

I didn't dig into quickbms code, but it may happen if you'll use space as delimiter for detecting pre-defined value in a string, or something. For example, you have [0x402C6A32 extracted\test file.bin] and you need to know, if it contains pre-defined value or not - in that case you search for the first space and split the string into pre-defined value and path. But what if you have only path with a space - such string will be splited in the same way (though it shouldn't). It's just a guess though.

[Davida]

Could you take a look at the format on X360 Far Cry 4 archives? This BMS script doesn't seem to work with them.
I'd be really thankful if you could release an updated script for this!

Sample file, 6MB: https://ufile.io/t5w8iy65

[h3x3r]

@Davida where did you get that *.NFO file? Beacause it contains filenames + hashed version. Post all of them if there are any others.

[FaisedWorld]

Hey, can somebody help me with reimporting The Crew 1 .dat files? I figured out the basics of exporting and reimporting but something doesn't seem right when reimporting a single file in a .dat file. I tried different script versions (0.1.5 / 0.1.6) and they give different results, none of them working.

0.1.5 seems to work but at the end of the reimported file there seems to be a spam of "DC1|NUL|NUL|NUL|NUL|NUL|NUL..." and so on, game doesn't open and crashes, while on 0.1.6 the entire reimported file is "DC1|NUL|NUL|NUL|NUL...", I'm using notepad++ to view btw.

The file I'm trying to mess with is from startup.dat from the original 2014 The Crew files. To be specific I'm trying to change a single 165KB xml file called "00000042.xml" with 0.1.5 or "683eecf8683eecf8" with 0.1.6, which is the network / server config file. It seems to try to connect to an older IP compared to the up-to-date TC1 startup.dat and I'm trying to make the original 2014 crew connect to the current TC1 servers in hopes that maybe I can get past the title screen and the #0_1 error it gives me when trying to enter the game. So basically I'm just trying to see if I can somehow play the original crew, which I know is unlikely but still, the least I can do is try because I really want to.

(If anyone's curious about the specific IPs: the up-to-date crew connects to 185.38.20.70:4000 when pressing any key on title screen, while the 2014 crew connects to 194.169.249.154:4000 which is outdated)


If anyone can help me with this reimporting problem then please do
Thanks!