Help finding compression method used

[xrubk]

Hi,

I'm trying to extract the data from a game file, the game file is compressed using zlib, and currently I was able to extract the file contents without any issue..

But now, it seems that the content of the extracted files is compressed, or encrypted somehow.

This is the file structure that I was able to extract from the game file:
Test_A792CBE3-D932-4832-A548-61E3F84D87E3 (folder)

• logo.jpg (an image that I'm able to add in game)
• _data (folder)
  
  • details.aom

As this gamefile has an image, that I can choose (ingame), this might help finding what compression method is being used.

So, in this case, the image that I've used, is the zenhax.com logo

Files attached:
Game file

Test_A792CBE3-D932-4832-A548-61E3F84D87E3.zip

Original logo.jpg file

logo.jpg

I've also used signsrch to see if it could help, here is the output:

Code: Select all

  offset   num  description [bits.endian.size]
  --------------------------------------------
  005cba05 3048 DMC compression [32.le.16&]
  01581ff8 1036 SHA1 / SHA0 / RIPEMD-160 initialization [32.le.20&]
  0158200d 2053 RIPEMD-128 InitState [32.le.16&]
  01c31b31 3051 compression algorithm seen in the game DreamKiller [32.be.12&]
  01c85aaa 2249 TEA1_DS [32.le.4]
  01e69733 2610 RNG [32.le.8&]
  02131374 1018 MD5 digest [32.le.272&]
  02169417 876  SHA256 Initial hash value H (0x6a09e667UL) [32.le.32&]
  021c65d2 948  DES [32.le.20&]
  0243a0cb 1299 classical random incrementer 0x343FD 0x269EC3 [32.le.8&]
  025afab8 641  CRC-32-IEEE 802.3 [crc32.0x04c11db7 le rev int_min.1024]
  025afab8 648  CRC-32-IEEE 802.3 [crc32.0xedb88320 lenorev 1.1024]
  025afeb8 129  Adler CRC32 (0x191b3141) [32.le.1024]
  025b02b8 131  Adler CRC32 (0x01c26a37) [32.le.1024]
  025b06b8 133  Adler CRC32 (0xb8bc6765) [32.le.1024]
  025b0ab8 645  CRC-32-IEEE 802.3 [crc32.0x04c11db7 be rev int_min.1024]
  025b0ab8 652  CRC-32-IEEE 802.3 [crc32.0xedb88320 benorev 1.1024]
  025b0eb8 130  Adler CRC32 (0x191b3141) [32.be.1024]
  025b12b8 132  Adler CRC32 (0x01c26a37) [32.be.1024]
  025b16b8 134  Adler CRC32 (0xb8bc6765) [32.be.1024]
  025b1ae8 2289 zinflate_lengthStarts [16.le.58]
  025b1b68 2296 zinflate_distanceStarts [16.le.60]
  025b1be8 1087 Zlib length_code [..256]
  025b1ce8 1086 Zlib dist_code [..512]
  025b1ee8 2294 zinflate_lengthExtraBits [32.le.116]
  025b1f70 2303 zinflate_distanceExtraBits [32.le.120]
  025b2530 1089 Zlib base_length [32.le.116]
  025b25a8 1091 Zlib base_dist [32.le.120]
  025b3758 3036 unlzx table_three [16.le.32]
  025c1498 1605 Generic bitmask table [32.le.128]
  025c1498 3038 unlzx table_three [32.le.64]
  025c150f 3050 compression algorithm seen in the game DreamKiller [32.le.12&]
  025cca98 1996 rfc3548 Base 64 Encoding with URL and Filename Safe Alphabet [..62]
  025ccb60 2005 B64EncodeTable [..64]
  025d2bb1 2417 MBC2 [32.le.248&]
  025d2bb4 2418 MBC2 [32.be.248&]
  02672e60 871  ACSS reverse sbox [..256]
  02672e60 1963 FFT and FHT routines rv_tbl [..128]
  02672f60 171  CRC-1 [crc8.0x01 lenorev int_min.256]
  02672f60 173  CRC-1 [crc8.0x01 le rev int_min.256]
  026744cc 1102 possible mask reset used in arithmetic compression [..9]
  026a3ac8 2065 Haval init [32.le.32&]
  026a3ac8 919  Blowfish bfp table [32.le.72]
  026a3ae8 1054 Haval hash pass2 [32.le.128&]
  026a3b20 878  Hash constant words K for SHA-384 and SHA-512 [64.le.640]
  026a3da0 874  SHA256 Hash constant words K (0x428a2f98) [32.le.256]
  026a3ea0 921  Blowfish ks0 table [32.le.1024]
  026a3ea0 2335 Blowfish_s_init [32.le.4096]
  026a3ef8 2067 Haval mc3 [32.le.128]
  026a3f58 2219 HAVAL2_DS [32.le.32]
  026a3f78 2069 Haval mc4 [32.le.128]
  026a3fd8 2217 HAVAL1_DS [32.le.32]
  026a3ff8 2071 Haval mc5 [32.le.128]
  026a42a0 923  Blowfish ks1 table [32.le.1024]
  026a46a0 925  Blowfish ks2 table [32.le.1024]
  026a4aa0 927  Blowfish ks3 table [32.le.1024]
  026a4f00 894  AES Rijndael S / ARIA S1 [..256]
  026a4fe0 2233 RIJNDAEL1_DS [..33]
  026a5000 914  Rijndael rcon [32.le.40]
  026a5028 895  AES Rijndael Si / ARIA X1 [..256]
  026a8a10 650  CRC-32-IEEE 802.3 [crc32.0xedb88320 le rev 1.1024]
  026a8a10 639  CRC-32-IEEE 802.3 [crc32.0x04c11db7 lenorev int_min.1024]
  026a9324 2588 bitmask [32.le.128]
  026a95c8 1931 libtheora old FrequencyCounts_VP3 [32.le.10240]
  026ade48 2875 libavcodec ff_mjpeg_val_ac_luminance [..162]
  026ae020 2876 libavcodec ff_mjpeg_val_ac_chrominance [..162]
  026afe0f 949  DES [32.be.20&]
  026b0368 1809 libdjvu GPixmap dither table [..256]
  026b0468 1115 Jpeg dct 14 bit aanscales [16.le.128]
  026b04e8 1119 Jpeg dct AA&N scale factor [double.le.64]
  026c6cd8 977  EC curve _EC_X9_62_PRIME_239V1_SEED [..20]
  026c6eb0 981  EC curve _EC_SECG_PRIME_112R1_SEED [..20]
  026c6f28 1001 EC curve _EC_X9_62_CHAR2_163V3_SEED [..20]
  026c7208 998  EC curve _EC_NIST_CHAR2_571B_SEED [..20]
  026c7510 987  EC curve _EC_SECG_CHAR2_113R1_SEED [..20]
  026c7590 971  EC curve _EC_NIST_PRIME_192_SEED [..20]
  026c7700 985  EC curve _EC_SECG_PRIME_160R1_SEED [..20]
  026c77d8 972  EC curve _EC_NIST_PRIME_224_SEED [..20]
  026c7c50 994  EC curve _EC_SECG_CHAR2_193R2_SEED [..20]
  026c7d28 982  EC curve _EC_SECG_PRIME_112R2_SEED [..20]
  026c7da0 979  EC curve _EC_X9_62_PRIME_239V3_SEED [..20]
  026c7e78 1008 EC curve _EC_X9_62_CHAR2_359V1_SEED [..20]
  026c7fe8 993  EC curve _EC_SECG_CHAR2_193R1_SEED [..20]
  026c8230 986  EC curve _EC_SECG_PRIME_160R2_SEED [..20]
  026c8570 980  EC curve _EC_X9_62_PRIME_256V1_SEED [..20]
  026c87f8 1005 EC curve _EC_X9_62_CHAR2_239V1_SEED [..20]
  026c8b18 1003 EC curve _EC_X9_62_CHAR2_191V2_SEED [..20]
  026c91c8 989  EC curve _EC_SECG_CHAR2_131R1_SEED [..20]
  026c9270 975  EC curve _EC_X9_62_PRIME_192V2_SEED [..20]
  026c9488 1007 EC curve _EC_X9_62_CHAR2_239V3_SEED [..20]
  026c9a10 978  EC curve _EC_X9_62_PRIME_239V2_SEED [..20]
  026c9c48 988  EC curve _EC_SECG_CHAR2_113R2_SEED [..20]
  026c9cc8 997  EC curve _EC_NIST_CHAR2_409B_SEED [..20]
  026c9e40 990  EC curve _EC_SECG_CHAR2_131R2_SEED [..20]
  026c9ee0 973  EC curve _EC_NIST_PRIME_384_SEED [..20]
  026ca1a8 1006 EC curve _EC_X9_62_CHAR2_239V2_SEED [..20]
  026ca280 976  EC curve _EC_X9_62_PRIME_192V3_SEED [..20]
  026ca350 995  EC curve _EC_NIST_CHAR2_233B_SEED [..20]
  026ca5e8 996  EC curve _EC_NIST_CHAR2_283B_SEED [..20]
  026ca898 1004 EC curve _EC_X9_62_CHAR2_191V3_SEED [..20]
  026ca968 974  EC curve _EC_NIST_PRIME_521_SEED [..20]
  026cac48 1000 EC curve _EC_X9_62_CHAR2_163V2_SEED [..20]
  026cadc8 984  EC curve _EC_SECG_PRIME_128R2_SEED [..20]
  026cae68 999  EC curve _EC_X9_62_CHAR2_163V1_SEED [..20]
  026cb140 983  EC curve _EC_SECG_PRIME_128R1_SEED [..20]
  026cb1e0 1002 EC curve _EC_X9_62_CHAR2_191V1_SEED [..20]
  026ce098 1009 ASCII to BIN table [..128]
  026d0550 900  Rijndael Te2 (0x63a5c663U) [32.le.1024]
  026d0950 911  Rijndael Td3 (0xf4a75051U) [32.le.1024]
  026d0d50 907  Rijndael Td1 (0x5051f4a7U) [32.le.1024]
  026d1178 898  Rijndael Te1 (0xa5c66363U) [32.le.1024]
  026d1578 896  Rijndael Te0 (0xc66363a5U) [32.le.1024]
  026d1978 902  Rijndael Te3 (0x6363a5c6U) [32.le.1024]
  026d1d78 905  Rijndael Td0 (0x51f4a750U) [32.le.1024]
  026d2278 909  Rijndael Td2 (0xa75051f4U) [32.le.1024]
  026d2678 1021 RC2 skey key_table [..256]
  026d27b8 950  DES odd_parity [..256]
  026d28d8 951  DES semi weak keys [..96]
  026d2938 952  DES skb [32.le.2048]
  026d29b8 2229 DES2_DS [32.le.128]
  026d3138 954  DES SPR SPtrans [32.le.2048]
  026d3938 2152 seed_SS0 [32.le.1024]
  026d3d38 2154 seed_SS1 [32.le.1024]
  026d4138 2156 seed_SS2 [32.le.1024]
  026d4538 2158 seed_SS3 [32.le.1024]
  026d4972 1947 small prime numbers used in libgcrypt [16.le.1336]
  026d703d 2079 Generic squared map [32.be.64]
  026d7040 2078 Generic squared map [32.le.64]
  026d8490 2125 CAST128 / CAST256 sbox3 [32.le.1024]
  026d8890 2127 CAST128 / CAST256 sbox4 [32.le.1024]
  026d8c90 2123 CAST128 / CAST256 sbox2 [32.le.1024]
  026d9090 2121 CAST128 / CAST256 sbox1 [32.le.1024]
  026d9490 2113 CAST128 sbox5 [32.le.1024]
  026d9890 2119 CAST128 sbox8 [32.le.1024]
  026d9c90 2117 CAST128 sbox7 [32.le.1024]
  026da090 2115 CAST128 sbox6 [32.le.1024]
  026da490 937  camellia [32.le.48&]
  026da4c0 939  camellia_sp1110 [32.le.1024]
  026da8c0 945  camellia_sp4404 [32.le.1024]
  026dacc0 941  camellia_sp0222 [32.le.1024]
  026db0c0 943  camellia_sp3033 [32.le.1024]
  026dcf08 1051 Whirlpool rc [64.be.80]
  026f1608 2298 zinflate_distanceStarts [32.le.120]
  026f27c8 2291 zinflate_lengthStarts [32.le.116]
  0276a0f0 915  Rijndael rcon [32.be.40]
  0276a1a8 882  Initial hash value H for SHA-512 [64.le.64&]
  02785e40 1290 __popcount_tab (compression?) [..256]
  028be00a 1289 Windows CryptDecrypt [..13]
  028be02c 1284 Windows CryptCreateHash [..16]
  028bebe6 2545 anti-debug: IsDebuggerPresent [..17]
  028c4d19 1038 padding used in hashing algorithms (0x80 0 ... 0) [..64]
  028c5bd0 2973 libavcodec sp5x_data_dht [..420]
  028c5f00 1113 Bzip2 BZ2_rNums [32.le.2048]

- 153 signatures found in the file in 10 seconds

[aluigi]

So:
- it's not compressed because the size of the new file is bigger
- there is a header of 0x28 bytes (probably 4[length1] + 4[length2] + length1 + length2]
- the original file wasn't aligned and the same is valid for the new file so it's not a block cipher (except if byte stream mode is selected which is rare for game archives)
- it's not xor with fixed key

It may be the RC4 encryption or a custom obfuscation.

Name of the game?

[xrubk]

the game is football manager 2016

[aluigi]

How you created that file?

[xrubk]

How you created that file?

It's a shortlist (list of players) that you can export from the game..

When you export your shortlist, you can choose an image that will be used by the game as a thumbnail for that shortlist..
(on my previous example, that's where i used a jpg version of zenhax logo)

In game:
Add a player to your shortlist, to do so:

• Navigate to some random player (not from your club)
• Menu Transfer (Top Menu) -> Add to Shortlist -> Any option..

Then to export your shortlist:

• Scouting(Left Menu) -> Shortlist (Top menu)
• Shortlist dropdown (Bottom ) -> Save shortlist as
• Click 'Change Preview Picture' to add your own image to the shortlist
• Save the shortlist

That will create a .fmf file that I already know how to extract (you can use this quickbms script to do so):

extract_script_bms.zip

[aluigi]

Here we go
http://aluigi.org/bms/fifa_manager_2016_shortlist.bms

I'm also updating my fmf script because there are some archives in the demo that don't work, I will post the update later.

[aluigi]

Unfortunately doesn't seem to exist a way for recognizing if an fmf archive is FM2014 or FM2016 so I had to make a new script which is just like the modified one you posted:
http://aluigi.org/bms/fifa_manager_2016.bms
I like the all-in-one scripts but not this time

[xrubk]

dude you're awesome!!

How did you find this so quickly? can you share some tips?

Cheers

[aluigi]

offbreak (offset 0x3a) + some tests with quickbms aes + luck.

I noticed that there was a xoring of the bytes of the file with a 16 bytes key that was regenerated every 16 bytes (one of the methods used for block ciphers applied to streams instead of blocks) and the generation code used AES tables with the x86 AES instructions (like AESENCLAST), some breakpoints in the code that uses the AES Te0 tables showed that the 2 sequence of 16 bytes at the beginning of the file where clearly involved in the initialization (arguments of the function).

So I did some tests with quickbms the normal AES encryption and the first sequence of XOR bytes matched, then I tried the aes_256_ctr algorithm without much hope it worked (I have ever had bad experiences with streaming block ciphers) and instead I had success at first shot.

offbreak has been created just for situations like this where it's necessary to find the code that works with the bytes of the file without doing statical analysis and wasting time: run game, debugger popups, breakpoint on data and you have the location of the decryption.
In fact most of the unknown compressions/encryptions of the unresolved topics in the forum can be probably solved quickly by using it.

[xrubk]

Very good explanation, thanks