My first quickBMS script

[aluigi]

Listen, I don't have the file. I have only a bounch of minutes to read your post and giving a quick feedback.
For sure the "goto -0x18" in the loop is wrong because it returns ever at the same point.

[AnonBaiter]

Well, that about clears up the confusion.
Besides, to not waste anyone's time, I'll be providing a sample instead. Be warned though, because the .BSA file has an early version compared to the one I'm working on(Battlespire).

EDIT: Here.

[aluigi]

Code: Select all

get FILES short
savepos OFFSET
get INFO_OFF asize
xmath INFO_OFF "INFO_OFF - (FILES * (14 + 4))"
goto INFO_OFF
for i = 0 < FILES
    getdstring NAME 14
    get SIZE long
    log NAME OFFSET SIZE
    math OFFSET + SIZE
next i

[AnonBaiter]

Umm, thanks mate.

EDIT: I just found out that this script can also be used on .BS6 files. As in, ones whose structure are strikingly similar to the BSA archive file. But it's not saying much, really.

[AnonBaiter]

So I'm trying to figure out a .DAT file. I want to find the offset/size based on these blocks:

Code: Select all

00000000  10 00 00 00 10 00 00 00 00 00 00 00 02 00 00 00  ................
00000010  10 00 00 00 10 00 00 00 00 00 00 00 01 00 00 00  ................
00000020  10 00 00 00 10 00 00 00 00 00 00 00 04 00 00 00  ................
00000030  02 00 00 00 A0 05 00 00 00 00 00 00 00 00 00 00  .... ...........

The problem though is I'm not sure if these are alignment for the files or something else, but then again I'm planning on writing a script that can find the offsets of the files based on these blocks.

Also, on the other .DAT file, there are folders without any reference to the files themselves:

Code: Select all

00000000  1D B2 C0 3B 01 00 01 00 15 00 10 40 01 00 00 00  .²À;.......@....
00000010  6E 6F 64 65 00 00 00 00 01 00 00 00 70 68 6F 74  node........phot
00000020  6F 00 00 00 5E 01 00 00 63 61 70 74 75 72 65 00  o...^...capture.
00000030  5B 02 00 00 6D 6F 62 69 6C 65 00 00 81 02 00 00  [...mobile......
00000040  66 30 30 61 00 00 00 00 97 02 00 00 66 30 31 61  f00a....—...f01a
00000050  00 00 00 00 C0 04 00 00 66 30 31 62 00 00 00 00  ....À...f01b....
00000060  D6 08 00 00 66 30 31 63 00 00 00 00 9E 0B 00 00  Ö...f01c....ž...
00000070  66 30 31 64 00 00 00 00 4B 0F 00 00 66 30 31 65  f01d....K...f01e
00000080  00 00 00 00 B0 12 00 00 66 30 31 66 00 00 00 00  ....°...f01f....
00000090  0D 17 00 00 66 30 32 61 00 00 00 00 C3 1A 00 00  ....f02a....Ã...
000000A0  66 30 33 61 00 00 00 00 3F 1D 00 00 66 30 33 62  f03a....?...f03b
000000B0  00 00 00 00 61 20 00 00 66 30 34 61 00 00 00 00  ....a ..f04a....
000000C0  12 22 00 00 66 30 34 62 00 00 00 00 1A 26 00 00  ."..f04b.....&..
000000D0  66 30 34 63 00 00 00 00 71 2A 00 00 66 30 34 64  f04c....q*..f04d
000000E0  00 00 00 00 A9 2E 00 00 66 30 34 65 00 00 00 00  ....©...f04e....
000000F0  84 32 00 00 66 30 35 61 00 00 00 00 BC 36 00 00  „2..f05a....¼6..
00000100  66 30 36 61 00 00 00 00 AB 38 00 00 00 00 00 00  f06a....«8......
00000110  00 00 00 00 00 00 00 00 DD 0D 67 8A 65 63 6D 3A  ........Ý.gŠecm:

However it does reference the offset of the filelist of these folders, and the filelist of a folder is as follows:

Code: Select all

00000800  1B 00 00 00 02 00 00 7F 40 D8 0A 00 D8 CA 6A 13  ........@Ø..ØÊj.
00000810  00 00 00 00 D5 23 9A 13 80 41 00 00 05 B4 99 13  ....Õ#š.€A...´™.
00000820  00 A8 02 00 0B 25 74 13 00 55 03 00 DC 57 77 13  .¨...%t..U..ÜWw.
00000830  00 C1 07 00 47 BE 01 13 00 23 08 00 D8 CA 6A 0E  .Á..G¾...#..ØÊj.
00000840  80 64 08 00 92 A6 A2 0E D0 76 08 00 24 BB 30 0E  €d..’¦¢.Ðv..$»0.
00000850  B0 7A 08 00 B3 B9 E4 0E C0 84 08 00 47 69 BB 0E  °z..³¹ä.À„..Gi».
00000860  C0 B8 08 00 73 41 C1 0E C0 EB 08 00 D5 23 9A 0E  À¸..sAÁ.Àë..Õ#š.
00000870  C0 1E 09 00 05 B4 99 0E 20 7A 09 00 17 CC CC 0E  À....´™. z...ÌÌ.
00000880  50 11 0A 00 47 BE 01 0E E0 86 0A 00 A4 AA 01 0E  P...G¾..à†..¤ª..
00000890  F0 AE 0A 00 AF 47 A4 0E 10 BC 0A 00 13 C9 34 0E  ð®..¯G¤..¼...É4.
000008A0  B0 CB 0A 00 12 C9 34 0E 10 CD 0A 00 11 C9 34 0E  °Ë...É4..Í...É4.
000008B0  70 CE 0A 00 D3 2D 33 0E D0 CF 0A 00 D2 2D 33 0E  pÎ..Ó-3.ÐÏ..Ò-3.
000008C0  A0 D2 0A 00 D1 2D 33 0E 70 D5 0A 00 00 00 00 7F   Ò..Ñ-3.pÕ......
000008D0  40 D8 0A 00 00 00 00 00 00 00 00 00 0E 46 0C 1B  @Ø...........F..
000008E0  16 B5 D4 51 2F 0D DD 8D 0C 4F 1B 5C 25 B6 38 85  .µÔQ/.Ý..O.\%¶8…

Anyway, here's the script for that .DAT file I'm writing on(sample comes later):

Code: Select all

get XOR long
get DUMMY1 long
get FOLDERS short
get DUMMY2 short
get DUMMY3 long

for FOLDER = 0 < FOLDERS
   getdstring NAME 0x8
   get OFFSET long
   math OFFSET * 0x800
   
   savepos TMP
   goto OFFSET
   
   string NAME + /
   
   get FILES long
   get DUMMY long
   math FILES - 3
   for i = 0 < FILES
      get OFFSET long
      get SIZE long
      math SIZE / 0x1000
      putarray 0 i OFFSET
      putarray 1 i SIZE
      log NAME OFFSET SIZE
   next i
   
   goto TMP
next FOLDER

The game in which these .DAT files came from is Metal Gear Solid 2(original PS2 version).

[AnonBaiter]

And here is an attached report of the script running in debug mode with an un-XORed file(FACE.DAT).
The second file only starts with the "0x00000000" offset, and when the script detects the files inside each folder it doesn't continue with the position that's been proposed(goto OFFSET). Rather, it goes right through the entire archive. Besides, I've been trying to come up with an solution for this(math OFFSET * 0x10) but what can I do?

[AnonBaiter]

And now for something completely different...

Code: Select all

get FILES long

math LAST_DISC_NUM = -1
math LAST_PART_NUM = -1
for i = 0 < FILES
   getdstring NAME 0xC
   get ZERO long
   get OFFSET long
   get SIZE long
   math OFFSET * 0x800
   math SIZE * 0x800
#   putarray 0 i NAME # for testing purposes
#   putarray 1 i OFFSET
#   putarray 2 i SIZE
   
   get TMP filename
   if PART_NUM != LAST_PART_NUM
      math LAST_PART_NUM == PART_NUM
      string TMP p "disc%d_%d.dat" DISC_NUM PART_NUM
   endif
   putarray 0 i TMP
   open FDSE TMP 1
   
   log NAME OFFSET SIZE
next i

So far it loaded only the first file(disc1_0.dat, disc2_0.dat), but it refuse to load the files that have the following numbers in their filenames(from disc1_1.dat to disc1_8.dat). Is there any solutions for this or has my question already get answered and I didn't pay attention to it?

[aluigi]

You should remove that "if PART_NUM == 0" condition.

[AnonBaiter]

Well, I did what you said and...

Code: Select all

- current_folder: E:\quickbms
- bms_folder:     E:\MGS2_SUBSTANCE
- exe_folder:     E:\quickbms
- file_folder:    E:\MGS2_SUBSTANCE\XBOX
- output_folder:  E:\quickbms
- temp_folder:    C:\Users\TCA\AppData\Local\Temp\
- open input file E:\MGS2_SUBSTANCE\XBOX\disc1_0.dat
- open script E:\MGS2_SUBSTANCE\mgs2_xbox.bms
- set output folder .

  offset   filesize   filename
--------------------------------------
. 00000000 get     FILES      0x00000004 4
. 00000004 getdstr NAME       "vox.dat" 12
    76 6f 78 2e 64 61 74 00 00 00 00 00               vox.dat.....
. 00000010 get     ZERO       0x00000000 4
. 00000014 get     OFFSET     0x00022ea0 4
. 00000018 get     SIZE       0x000a2960 4
. 0000001c get     TMP        "disc1_0.dat" -1000
. 00000000 putarr  TMP        "disc0_0.dat" 0:0
- enter in folder E:\MGS2_SUBSTANCE\XBOX
- open input file E:\MGS2_SUBSTANCE\XBOX\disc0_0.dat

- error in src\file.c line 243: fdnum_open()
Error: No such file or directory

Last script line before the error or that produced the error:
  22  open FDSE TMP 1

As you can see, I'm using two variables: one for the disc number("disc%d"), and one for the part number(_%d).
So I want the script to start detecting the "disk" archive files from the first number(1).

Maybe I'm just that incompetent...

[aluigi]

Ah now I get it, you check a PART_NUM variable that doesn't exist.
Same for DISC_NUM.
If you don't have these variables how you expect the script to open something?

[AnonBaiter]

Well, you have a point there. Silly me.
I'll think of another way to have these variables, but I'm afraid I'll just be running out of ideas(and sometimes overall competence) at this point. I'm open to all suggestions at the moment.

[aluigi]

Well, a suggestion... you have a ZERO field there, does it remain ZERO for all the entries or does it change?

[AnonBaiter]

The ZERO field was only added to make sure the "getdstring NAME 0xC" doesn't get in the way of detecting the wrong filenames(such as c.dat if set to "set NAME string"), so it remains unchanged for all entries. See here:

Code: Select all

- current_folder: E:\quickbms
- bms_folder:     E:\MGS2_SUBSTANCE
- exe_folder:     E:\quickbms
- file_folder:    E:\MGS2_SUBSTANCE\XBOX
- output_folder:  E:\quickbms
- temp_folder:    C:\Users\TCA\AppData\Local\Temp\
- open input file E:\MGS2_SUBSTANCE\XBOX\disc1_0.dat
- open script E:\MGS2_SUBSTANCE\mgs2_xbox.bms
- set output folder .

  offset           filesize   filename
--------------------------------------
. 0000000000000000 get     FILES      0x0000000000000004 4
. 00000000 putarr  LAST_DISC_NUM 0xffffffffffffffff 0:0
. 00000000 putarr  LAST_PART_NUM 0xffffffffffffffff 1:0
. 0000000000000004 getdstr NAME       "vox.dat" 12
    76 6f 78 2e 64 61 74 00 00 00 00 00               vox.dat.....
. 0000000000000010 get     ZERO       0x0000000000000000 4
. 0000000000000014 get     OFFSET     0x0000000000022ea0 4
. 0000000000000018 get     SIZE       0x00000000000a2960 4
  0000000011750000 1363869696 vox.dat
.
. 0000000000000001
. 000000000000001c getdstr NAME       "codec.dat" 12
    63 6f 64 65 63 2e 64 61 74 00 00 00               codec.dat...
. 0000000000000028 get     ZERO       0x0000000000000000 4
. 000000000000002c get     OFFSET     0x00000000000c5800 4
. 0000000000000030 get     SIZE       0x0000000000006b78 4
  0000000062c00000 56344576   codec.dat
.
. 0000000000000002
. 0000000000000034 getdstr NAME       "face.dat" 12
    66 61 63 65 2e 64 61 74 00 00 00 00               face.dat....
. 0000000000000040 get     ZERO       0x0000000000000000 4
. 0000000000000044 get     OFFSET     0x00000000000cc378 4
. 0000000000000048 get     SIZE       0x0000000000003224 4
  00000000661bc000 26288128   face.dat
.
. 0000000000000003
. 000000000000004c getdstr NAME       "stage.dat" 12
    73 74 61 67 65 2e 64 61 74 00 00 00               stage.dat...
. 0000000000000058 get     ZERO       0x0000000000000000 4
. 000000000000005c get     OFFSET     0x00000000000cf59c 4
. 0000000000000060 get     SIZE       0x00000000000cab3d 4
  0000000067ace000 1700390912 stage.dat
.
. 0000000000000004

- 4 files found in 0 seconds
  coverage file 0     0%   100        187420672

---

- current_folder: E:\quickbms
- bms_folder:     E:\MGS2_SUBSTANCE
- exe_folder:     E:\quickbms
- file_folder:    E:\MGS2_SUBSTANCE\XBOX
- output_folder:  E:\quickbms
- temp_folder:    C:\Users\TCA\AppData\Local\Temp\
- open input file E:\MGS2_SUBSTANCE\XBOX\disc2_0.dat
- open script E:\MGS2_SUBSTANCE\mgs2_xbox.bms
- set output folder .

  offset           filesize   filename
--------------------------------------
. 0000000000000000 get     FILES      0x0000000000000003 4
. 00000000 putarr  LAST_DISC_NUM 0xffffffffffffffff 0:0
. 00000000 putarr  LAST_PART_NUM 0xffffffffffffffff 1:0
. 0000000000000004 getdstr NAME       "demo.dat" 12
    64 65 6d 6f 2e 64 61 74 00 00 00 00               demo.dat....
. 0000000000000010 get     ZERO       0x0000000000000000 4
. 0000000000000014 get     OFFSET     0x00000000000641e8 4
. 0000000000000018 get     SIZE       0x00000000000bc982 4
  00000000320f4000 1582043136 demo.dat
.
. 0000000000000001
. 000000000000001c getdstr NAME       "movie.dat" 12
    6d 6f 76 69 65 2e 64 61 74 00 00 00               movie.dat...
. 0000000000000028 get     ZERO       0x0000000000000000 4
. 000000000000002c get     OFFSET     0x0000000000120b6a 4
. 0000000000000030 get     SIZE       0x0000000000078c4e 4
  00000000905b5000 1013084160 movie.dat
.
. 0000000000000002
. 0000000000000034 getdstr NAME       "movievr.dat" 12
    6d 6f 76 69 65 76 72 2e 64 61 74 00               movievr.dat.
. 0000000000000040 get     ZERO       0x0000000000000000 4
. 0000000000000044 get     OFFSET     0x00000000001997b8 4
. 0000000000000048 get     SIZE       0x00000000000015e8 4
  00000000ccbdc000 11485184   movievr.dat
.
. 0000000000000003

- 3 files found in 0 seconds
  coverage file 0     0%   76         624287744

[AnonBaiter]

All right, I decided to take another look at extracting MGS2 data by myself...
Anyway, what do you think about this script? Do you think it needs some work or something? Because personally I'm thinking about detecting the offset location of these files just by using these "findloc" functions...

Code: Select all

# for use with PS2 version only(not tested with Substance yet)
# only the Japanese version was used

get FULLSIZE asize
#findloc START_OFFSET binary "\x10\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00" # for DEMO.DAT(original PS2 release)
#findloc START_OFFSET binary "\x10\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00" # for VOX.DAT(original PS2 release/The Document of Metal Gear Solid 2)
#findloc START_OFFSET binary "\x10\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x0e\x00\x00\x00" # for MOVIE.DAT/MOVIE2.DAT(The Document of Metal Gear Solid 2)

for
   findloc START_OFFSET binary "\x10\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x0f\x00\x00\x00" # for MOVIE.DAT(original PS2 release)
   goto START_OFFSET
   if START_OFFSET >= FULLSIZE
      cleanexit
   endif
   savepos OFFSET
   findloc ALIGNMENT binary "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x0f\x00\x00\x00"
   findloc END_OFFSET string START_OFFSET 0 ""
   if END_OFFSET == ""
      math END_OFFSET = FULLSIZE
   endif
   math END_OFFSET = ALIGNMENT
   /*
   xmath SIZE "START_OFFSET == END_OFFSET"
   goto END_OFFSET
   padding 16
   do
      getdstring TMP 16
   while TMP == START_OFFSET
   */
   savepos TMP
   putarray 0 i START_OFFSET
   putarray 1 i END_OFFSET
   string NAME p "%04d." i
   putarray 2 i NAME
   getarray SIZE 0 i
   if SIZE != 0
      log NAME OFFSET SIZE
   endif
next

[aluigi]

Probably with that format it's better to just dump all the RIFF files.
This is what I adopted in script 0.1.3 to skip the errors.

[AnonBaiter]

Except that in these archives, there are no signs of "RIFF" floating around anywhere, which makes finding them harder than it should be.
So far I'm left with these signs:

Code: Select all

Original PS2 version:
"\x02\x00\x00\x80\x20" - DEMO.DAT
"ipu"\"ipum" - MOVIE.DAT
"\x00\x02\x10\x00"\"\x00\x00\xc0\x00" - VOX.DAT(each file has a varying number, these two are proof of this)
The Document of Metal Gear Solid 2:
"\x00\x00\x01\xBA\x44\x00\x04\x00" - MOVIE.DAT/MOVIE2.DAT
Metal Gear Solid 2 Substance(PS2):
Same as "Original PS2 version".
Metal Gear Solid 2 Substance(XBOX):
"\x00\x01\xc0\x00" - movie.dat
"\xb0\x00\x80\x00" - movievr.dat

Also, my script never goes beyond the first file no matter what I do. Perhaps it has to do with my inability with writing complex scripts or something...
(Nevermind I still don't know what to do by now, so I'll have to stick with dull hex strings like "1000000010000000000000000f000000")

[AnonBaiter]

So I decided to grab both versions of a PS1 game called Metal Gear Solid: the American version(two discs) and the Japanese Integral version(three discs; it served as the base of the inferior PC version sound-wise). So far I managed to write a script for the DEMO.DAT archive that was stored on both version I posess at hand. Here's the script:

Code: Select all

get FULLSIZE asize
findloc START_OFFSET binary "\x10\x08\x00\x00\x05\x00\x00\x00\x05"

for i = 0
   savepos OFFSET
   findloc ALIGNMENT binary "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x08\x00\x00"
   findloc END_OFFSET string START_OFFSET 0 ""
   if END_OFFSET == ""
      math END_OFFSET = FULLSIZE
   endif
   math END_OFFSET x ALIGNMENT
   /*
   if ALIGNMENT != ""
      cleanexit
   endif
   */
   putarray 0 i START_OFFSET
   putarray 1 i END_OFFSET
   math SIZE = END_OFFSET
   math SIZE - OFFSET
   string NAME p "%04d.dmo" i
   if SIZE != 0
        log NAME OFFSET SIZE
    endif
   goto END_OFFSET
   padding 16
   do
      getdstring TMP 16
   while TMP == START_OFFSET
next i

So far the only problem with this script is that it doesn't dump the last file; perhaps it has to do with the "ALIGNMENT" argument. If you want a sample of DEMO.DAT(with filecutter, no less) then you should just ask.

[aluigi]

There is for sure something wrong in that script
The pattern in START_OFFSET should be used directly in findloc, and there is something strange with the arrays.
Maybe check if the following works:

Code: Select all

for i = 0
    findloc OFFSET binary "\x10\x08\x00\x00\x05\x00\x00\x00\x05"
    goto OFFSET
    findloc END_OFFSET binary "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x08\x00\x00"
    math SIZE = END_OFFSET
    math SIZE - OFFSET
    string NAME p "%04d.dmo" i
    log NAME OFFSET SIZE
    goto END_OFFSET
    padding 16
next i

[AnonBaiter]

Well, it did work somewhat but that doesn`t solve the problem of the last file not being extracted.

Code: Select all

0EE32770  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE32780  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE32790  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE327A0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE327B0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE327C0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE327D0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE327E0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE327F0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0EE32800  10 08 00 00 05 00 00 00 05 9C 00 00 00 00 00 00  .........œ......
0EE32810  6B 14 00 00 01 00 00 00 06 00 00 00 1C 00 00 00  k...............
0EE32820  24 00 00 00 31 F4 0A 00 31 F4 00 00 01 00 00 00  $...1ô..1ô......
0EE32830  00 00 00 00 2D 99 0A 00 2D 99 00 00 00 00 00 00  ....-™..-™......
0EE32840  02 00 00 00 00 00 00 00 88 0A 0A 00 88 0A 00 00  ........ˆ...ˆ...
0EE32850  00 00 00 00 03 00 00 00 00 00 00 00 BF FF 0A 00  ............¿ÿ..
0EE32860  BF FF 00 00 00 00 00 00 04 00 00 00 00 00 00 00  ¿ÿ..............
0EE32870  B6 21 0A 00 B6 21 00 00 00 00 00 00 05 00 00 00  ¶!..¶!..........
0EE32880  00 00 00 00 3B 2D 0A 00 3B 2D 00 00 00 00 00 00  ....;-..;-......
0EE32890  06 00 00 00 00 00 00 00 59 E7 0A 00 59 E7 00 00  ........Yç..Yç..
0EE328A0  00 00 00 00 10 08 00 00 01 00 00 00 02 14 00 00  ................
0EE328B0  00 64 80 00 3F FF 0C 00 02 01 00 00 00 00 00 00  .d€.?ÿ..........
0EE328C0  10 08 00 00 03 00 00 00 04 08 00 00 01 00 00 00  ................
0EE328D0  03 04 06 00 00 00 00 00 FF FF FF 7F 10 00 14 00  ........ÿÿÿ.....
0EE328E0  00 06 00 00 F0 00 00 00 1C 00 00 00 71 00 00 00  ....ð.......q...

Just so you know, the last file is located at offset 0xee32800. Yet the file ends at the 0xf69f7ff offset, which END_OFFSET is incapable of handing because there are a lot of zeroes instead of the hex-strings said function/argument is assigned to.
What I'm saying is this: is it possible to add an exception if the script is going to dump the last file?

[aluigi]

It depends by what of the two findloc fails first.
Add the usual 0 "" to the findloc command and it will be able to handle the missing pattern.