Killzone (PS2) - .DAT files

[AnonBaiter]

This game was developed by Guerrilha Games, which was responsible for the game Shellshock: Nam '67 right before they were one of those Sony`s subsidiary companies.
Here are the samples.
EDIT: Anyway here's an script I attempted to write.

Code: Select all

open FDSE "FILES.DAT"
get TOTAL_SIZE long
get FILES long
get DUMMY1 long
get DUMMY2 long
get ZERO longlong
get REF_OFF long
math REF_OFF + 0x18
putarray 0 i REF_OFF

for i = 0 < FILES
   get NAME string
next i
goto REF_OFF
get ZERO long
for i = 0 < FILES
   get DUMMY1 long
   get DUMMY2 long
   get DUMMY3 long
   get DUMMY4 long
next i

[AnonBaiter]

Uhh... bump?

[aluigi]

*edit*

[aluigi]

Opss I confused it with another game.
Regarding this one... I don't know if and when I will check it.

[aluigi]

No idea where are OFFSET and SIZE, the following work-in-progress (USELESS, DOES NOT EXTRACT ANYTHING) is what I can recognize:

Code: Select all

get INFO_SIZE long
get NAMES long
get DUMMY3 long
get DUMMY4 long
get DUMMY5 long
get DUMMY6 long
get NAMES_SIZE long
savepos OFFSET
log MEMORY_FILE OFFSET NAMES_SIZE
math OFFSET + NAMES_SIZE
goto OFFSET
get FILES long
math FILES / 12
for i = 0 < FILES
    get DUMMY long
    get NAME_OFF threebyte
    get DUMMY byte
    get DUMMY long

    goto NAME_OFF MEMORY_FILE
    get NAME string MEMORY_FILE
next i

[AnonBaiter]

It's been a while, but this is what I got when I ran the script:

Code: Select all

.
. 0000000000000039
. 0000000000030dbc get     DUMMY      0x0000000000000048 4
. 0000000000030dc0 get     NAME_OFF   0x0000000000805fe2 3
. 0000000000030dc3 get     DUMMY      0x0000000000000001 1
. 0000000000030dc4 get     DUMMY      0x0000000015eee000 4

This is the error:

Code: Select all

QuickBMS generic files extractor and reimporter 0.7.7 (64bit test)
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Dec  3 2016 - 21:43:01)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- current_folder: E:\quickbms
- bms_folder:     E:\KILLZONE
- exe_folder:     E:\quickbms
- file_folder:    E:\KILLZONE
- output_folder:  E:\quickbms\cmd
- temp_folder:    C:\Users\TCA\AppData\Local\Temp\
- open input file E:\KILLZONE\FILES.DAT
- open script E:\KILLZONE\killzone.bms
- set output folder cmd

  offset           filesize   filename
--------------------------------------

Error: [myfseek] the offset 0x0000000000805fe2 in the file -1 can't be reached

Last script line before the error or that produced the error:
  20  goto NAME_OFF MEMORY_FILE

I'm guessing there are no fixes to this, right?

[Ekey]

No idea where are OFFSET and SIZE, the following work-in-progress (USELESS, DOES NOT EXTRACT ANYTHING)

[AnonBaiter]

So it's confirmed; the archives themselves are messy enough as it is.

[AnonBaiter]

Have you(aluigi) ever considered using getbits for the NAME_OFF variable?

from

Code: Select all

    get NAME_OFF threebyte
    get DUMMY byte

to

Code: Select all

   getbits NAME_OFF 20
   getbits DUMMY2 10

Not to say it's a logical solution, but what can I say?

[AnonBaiter]

Okay, it's double-post time.

What I can gather out of these DUMMY2(10-bit type) values are the following:

Code: Select all

0x00 value = FILES.DAT
0x08(08) value = FILES01.DAT
0x10(16) value = FILES02.DAT
0x18(24) value = FILES03.DAT
0x20(32) value = FILES04.DAT

Now divide these DUMMY2 values by 8. You can see that DUMMY2 is actually FILES_NUM.

Do be careful when there are DUMMY2/FILES_NUM values that end up greater than 4. In any case...

Code: Select all

    if FILES_NUM != LAST_FILES_NUM
        math LAST_FILES_NUM = FILES_NUM
        if FILES_NUM == 0
            string ARCHIVE_NAME p "files.dat"
      elif FILES_NUM > 4
         break
        else
            string ARCHIVE_NAME p "files%02d.dat" FILES_NUM
        endif
        open FDSE ARCHIVE_NAME 1
    endif

Another thing I should mention is that there is an OFFSET field. It references the offset of the files based on each part of the archive files and treating them as the located offsets in these individual archives rather than treating said archives as one and having the offset fields being referenced as being the one and only archive file.

Let's take this .mss file, for example:

Code: Select all

.
. 0000000000000039
. 0000000000030dbc get     DUMMY1     0x0000000000000048 4
. 0000000000030dc0 getbits NAME_OFF   0x0000000000005fe2 20
. 0000000000030dc3 getbits FILES_NUM  0x0000000000000018 10
. 0000000000030dc4 get     OFFSET     0x0000000015eee000 4
. 00000000 putarr  FILES_NUM  0x0000000000000003 0:57
. 0000000000005fe2 get     NAME       "extra_isa2_3925.mss" -1

An .mss file has an "MCSS" sign, followed by header information.
If we ever want to look out for that .mss file, then we need to look for the archive file that references the FILES_NUM part(FILES03.DAT) and then look out for the offset of that .mss file. The thing is, there is already an referenced offset for that, so by going to that offset in FILES03.DAT we already know where that .mss file is located. See where I'm going with this?

Here's an attached script which was based on your(aluigi) own work-in-progress script, and a -V -0 report for further reference.

[aluigi]

So how many PART*.PAK are used by your GOW2? 2 or 5?
The final exact number with any *hidden* pak.

[AnonBaiter]

So how many PART*.PAK are used by your GOW2? 2 or 5?
The final exact number with any *hidden* pak.

It`s two parts in the USA version. The "hidden" pak doesn't exist, after all I was mistaken.
As I said, the number of PART*.PAK files depends on the region the game was released on so (as of this writing) I only have that copy to check.

[aluigi]

I confused this thread with the other one of Gear of Wars...

[aluigi]

Script 0.2
http://aluigi.org/bms/killzone.bms

Anyway there are some strange things with the samples you provided:
- the references to FILES02.dat in reality are for FILES04.dat
- there are references only for FILES, FILES01 and FILES02, so the other 2 archives are out

Probably am I missing something in the information to parse?

*edit*
No, I'm missing nothing as stated by the coverage of the tables:
coverage file -1 118%
coverage file -2 100%

[AnonBaiter]

I confused this thread with the other one of Gear of Wars...

You mean Gears of War or God of War? No wonder you're so confused, sheesh.

Anyway, just now I tested your new update and it triggered this error(I didn't extract anything!):

Code: Select all

- current_folder: E:\quickbms
- bms_folder:     E:\KILLZONE
- exe_folder:     E:\quickbms
- file_folder:    E:\KILLZONE
- output_folder:  E:\quickbms\cmd
- temp_folder:    C:\Users\TCA\AppData\Local\Temp\
- open input file E:\KILLZONE\FILES.DAT
- open script E:\KILLZONE\killzone.bms
- set output folder cmd

  offset           filesize   filename
--------------------------------------
- enter in folder E:\KILLZONE
  coverage file 0     0%   352228     1013012484 . offset 0000000000030b10
- open input file E:\KILLZONE\FILES01.DAT

Error: the compressed zlib/deflate input is wrong or incomplete (-3)
Info:  algorithm   1
       offset      0000000008339008
       input size  0x00000000002dad5c 2993500
       output size 0x0000000044008e50 1140887120
       result      0xffffffffffffffff -1

Error: the uncompressed data (-1) is bigger than the allocated buffer (1140887120)

Last script line before the error or that produced the error:
  58  clog NAME OFFSET SIZE XSIZE

It happened as soon as it reached this file:

Code: Select all

. 00000000 getarr  LAST_PACK_NUM 0x0000000000000001 0:0
. 0000000000019bfc get     SIZE       0x00000000002dad64 4
. 0000000008339000 get     ZERO       0x00000000ba010000 4
. 0000000008339004 get     XSIZE      0x00000000508e0044 4
  0000000008339008 1140887120 /localcache.ps2/lumps/allweapons.default.core

Not to mention the offsets are a bit off(at least for those LAST_PACK_NUM archives), which means the .mss file extracted is not actually an .mss file at all...
Perhaps it's just my copy of the game that's affected, but I felt the need to post it here.

[aluigi]

Yeah God of War

I have updated the script to version 0.2.1 for enabling the automatic decompression of the "core" files only on PSP.dat

[AnonBaiter]

Okay, let's see if it actually works this time...

EDIT: Well...

Code: Select all

  000000000ac89800 126884     /localcache.ps2/lumps/entitysound.helghast.3rd.generic.english.core
  000000003af54000 4105536    /localcache.ps2/lumps/entitysound.helghast.3rd.generic.english.corestream

Error: incomplete input file 0: E:\KILLZONE\FILES01.DAT
       Can't read 700412 bytes from offset 000000003b0a9004.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0    31%   311682156  990547972  . offset 000000003b0a9004

Last script line before the error or that produced the error:
  60  log NAME OFFSET SIZE

What's causing the error? The wrong offset values or something related fo LAST_PACK_NUM?
I can't think of anything else really, especially since I have evidence for what is possibly causing the error to trigger all of a sudden. And I'm not even using -V -0 this time!
At least the PSP.DAT file works with the updated script.

[aluigi]

Script 0.2.2, this is the right time.
The FLAGS field is composed by the following bits:
1: folder/file
3: ???
5: package number
23: name offset

Now the script loads all the 5 packages.

[AnonBaiter]

Seems like I was right after all.

[aluigi]

bitfields are ever a pain.