reimport2/3: Possible to delete original data and update all of the file table?

[Risae]

Dear Forum,

a quick overview of the files that i am working with:

Growlanser 6.ISO
>GL6_SCEN.DAT
>>00000000.flk
>>00000001.flk...

i have been using quickBMS reimport1 for my PS2 translation project for quite a while now, it works without a flaw.
Now i am coming across an issue with space.
The english script that i am importing into the .FLK files are getting bigger than the original.
So, i was trying to use import2 and 3 to see if it can help me with the size restrictions using reimport1.
Thanks to the awesome Growlanser.bms script made by Sir Aluigi, i was able to use reimport2 on 1 file and put it at the end of the .DAT + automaticly update the file table at the start of the .DAT file:

https://i.imgur.com/yHfHDGq.png

Now here is the problem that i am currently facing.
It seems like if i go over a certain amount of filesize of GL6_SCEN.DAT, the game crashes inside PCSX2 and gives me various errors:

https://i.imgur.com/SW8cj7P.png

My current theory is that the "overflow" data gets put in a place where the game stores other data and my GL6_SCEN.DAT data gets cut off at some point.

I saw that the reimport2 only puts the data at the very end of the file and overwrites the original data with a bunch of 00s.
I tried to delete the 00s that were overwritten, but that seems to completely break the game too, since the position data at the beginning is not properly showing the start of each file anymore.

My question would be: How could i solve this issue?
Is it possible to, instead of writing the data at the very end of GL6_SCEN.DAT, to put it in its original position and "push" the other data down while also updating all the location data at the beginning of GL6_SCEN.DAT?

[Shokoniraya]

sure, it possible to do that. you talking about reimporting, right? pushing files to add more bytes in raw files required updating other file offets that is bigger than current file, so you should make a repacker based on original table or update any other important value that sets chunk sizes or archive size

there is many ways to do that, but without sample, no one can help you cause other guys don't have game

[Risae]

Hi Shokoniraya,

thank you very much for your reply!
Yes, i meant reimporting. A repacker? Could you please explain more what you meant with that? Is there a difference for a repacker to quickBMS reimport?

Oh, i apologize for not adding a sample file.
Here is the original GL6_SCEN.DAT + a folder containing the modified .FLK files which got over the original size and the ones that didn't go over:

https://anonfiles.com/F9k96f48of/GL6_Tr ... roject_rar

I used this quickBMS script:

http://aluigi.zenhax.com/bms/growlanser.bms

[Shokoniraya]

there is a TOTAL_ARCHIVE_SIZE in offset 20 (hex:0x14), i think that's why game breaks

[Risae]

Hi Shokoniraya!

Thank you so much for your script!
I have a question for the usage of your script:

- Is this script for reimport1, 2 or3?
- Should i put ALL of the original files inside the import folder, or can i import only the modified ones?

Again, thank you very much!


Edit: Ah, it seems like with your script the files are getting dumped in a different way.
I will input the modified scripts into the new output files and see if it works!


Edit2:
Okay, i input all of the scripts into the files and reimported them with quickBMS reimport2 into the .DAT.
I rebuild the ISO using Xpert but it seems like it still doesn't like the new DAT file:

https://i.imgur.com/xvG0hM6.png

[Shokoniraya]

just remeber that you must use -w in export, just like importing

[anakronus]

Analysing extracted flk files (from the .DATs) the HEADER_FLAG field is actually the amount of FILE_EXTENSIONs in the flk and PADDING_NUMBER is an offset to the end of the file offset table (or the start of the FILE_EXTENSIONs table if you want to see it that way), I assume when they are 0 the file is a package?

Also, UNKNOW_3 seems actually 2 shorts, first one is unknown (0 for the dats and 1 for extracted flks, so probably a flag) and the other one is the amount of FILE_EXTENSIONs in the flk (again).

And lastly, some DAT's have files beyond the last file inside of an extension, so they throw an error of going beyond the array, but excellent work on the script!

[Risae]

Hi guys,

first off, thank you very much Shokoniraya for that awesome script.

The MIPS hacker for the translation project also took a look at this issue, and he found out this:

I figured it out! After blankly staring at hex for too long I figured thisthing out.
Remember all those silly spaces at the end of every file? They are CRITICAL, all of them pad out the files for them to snuggly fit inside an
LBA (0x800 chunks that divide the game disc logically) the game will simply refuse to read in-between LBA's for speed (and because they have ~4GB to
work with anyways ) as such, if the file doesn't start at a disc location where 0xLocation % 0x800 == 0 the game will just discard the part that
isn't and will do direct copy starting from a location that is multiple of 0x800.

THAT'S why it breaks when the changes are big enough, but leaves a little wiggle room where if small enough it can work by pure luck, and that's also
why the defined .DAT size isn't really honoured.

Every .DAT file is like this and that explains why the .flk's inside were multiples of 2, now I haven't tested this yet, but a comparison of two
instances of the game running confirms this suspicion (and also confirms that the game can handle relocating bigger files, assuming enough memory is
available) so we only need to automate this.

And lastly some funny observation, I actually pointed at something like this in a previous email, past me basically had it resolved xd

Just tested, all working ok after padding the file manually.
[image: image.png]

A really nice person over at the quickBMS forums replied to my post too, and with his quickBMS script it seems like the dumping of the .DAT files is
different.

Oh wow, this guy is god, we got mostly the same understanding of the file
format, but he actually figured out more fields in less time, awesome!

I guess because of the broken file sizes (0xLocation % 0x800 == 0), which
affects a lot of files now, even with his script the game will not load the
files properly.

I'd say his script is pretty good, the issue now seems to be due the way
you dealt with the flk's (or rather STXT) that wasn't accounting for this
change in the actual file definitions, assuming they are "mishandled" in
the same way it is rather easy to fix

How many bytes would i need to add to the script file in order for the
game to be able to read it properly?

Thi is very simple, the file needs to end with 0x000 or 0x800

when i tried to load the endgame save the whole game was suuuuuper slow
(not inside the menu though) and crashes after opening the menu and closing
it

It seems that some files got offsetted by 0x80 in some places, it's
probably that offset the one causing the issues, can't check right now.

Anyway, checking Shokoniraya's very awesome script, it actually seems to do
most everything for remaking the .DAT file (this guy is god)
So, theoretically you would only need to make the STXT's a size multiple of
0x80 it seems, and ideally it should be just a script of some sort (as the
new STXT, are flk's with SDF files, which themselves have a format) to have
it in just one step.

Gotcha, Shokoniraya actually send me an updated version for the script:

Perfect! Now the files are padded to 2048 (0x800) it should work as-is now,
now it's time time to test it

Gotta say, quickbms is awesome considering its generic nature, this script
is awesome considering it made unnecessary to code a parser ourselves

I didn't have time to test the script yet, but i really appreciate all the help from you Shokoniraya!
I will update this thread once i tested it.


Edit: But just for my understanding, i'm using reimport2 for this script, right?

[Risae]

Holy crap, i think it works!
I exported all of the files inside SCEN.DAT using your script, input the scripts and then imported all files into the SCEN.DAT again and so far everything works.
No slowdown, no crashes, no strange behavior of the script.
I will do more tests later, but so far it looks like its working! Thank you so, so much Shokoniraya!

Since your script seems to name the files differently than the original growlanser.bms script, i tried to use your new script to export the other .DAT files the game has:

DUMMY.DAT
GL6_BTL.DAT
GL6_CHAR.DAT
GL6_FACE.DAT
GL6_FILE.DAT
GL6_MAP.DAT
GL6_MOV.DAT
GL6_SCEN.DAT
GL6_SND.DAT
GL6_VOI.DAT

but it seems like sometimes the script doesn't work properly.
Would it be okay if you could take a look if its possible to create a script to properly dump the other .DAT files of Growlanser 6?

https://anonfiles.com/b4vaMa46o2/GL6_DAT_Files_rar

By the way, Growlanser 5 JPN/ENG + Growlanser 6 JPN use the same engine with little to no changes to the files and its filestructure.
With the original growlanser.bms script i more or less was able to dump all of the files from GL5 and 6.
Since the SCEN script works on Growlanser 6, it should be possible to also work on the original Growlanser 5 SCEN.DAT files.

[Shokoniraya]

my network socket is really slow and I cant download this archive
can you please upload smallest file that script can't works with it?

[Risae]

Using the script on DUMMY.DAT gives the following error:
https://anonfiles.com/PfAcWd4do2/DUMMY_DAT

Code: Select all

Error: the script uses more array indexes (442) than available (442)

Last script line before the error or that produced the error:
  43  getarray FILE_EXTENSION 1 NUMBERIC_COUNT

Using the script on GL6_CHAR.DAT gives the following error:
https://anonfiles.com/NaB1Wa49of/GL6_CHAR_DAT

Code: Select all

Error: the script uses more array indexes (372) than available (372)

Last script line before the error or that produced the error:
  43  getarray FILE_EXTENSION 1 NUMBERIC_COUNT

Using the script on GL6_FACE.DAT didn't give any error
https://anonfiles.com/HdCaW144ob/GL6_FACE_DAT



Using the script on GL6_FILE.DAT gives the following error:
https://anonfiles.com/j6CcW047o7/GL6_FILE_DAT

Code: Select all

Error: the script uses more array indexes (564) than available (564)

Last script line before the error or that produced the error:
  43  getarray FILE_EXTENSION 1 NUMBERIC_COUNT

Using the script on GL6_MAP.DAT didn't give any error
https://anonfiles.com/veF3W844o8/GL6_MAP_DAT



Using the script on GL6_SND.DAT didn't give any error, but there are a lot of files that have 0 kb get exported.
https://anonfiles.com/v2H9W143o8/GL6_SND_DAT



Using the script on GL6_VOI.DAT didn't give any error
https://anonfiles.com/b6L9Wc46o1/GL6_VOI_DAT

[Shokoniraya]

script has update

[Risae]

Hi Shokoniraya,

thank you very much for your updated script!
I tried it, but it seems like normal files that should be named

0000002a.fnt
0000002b.fnt...

are now named

00000XXX.bin

And this is happening to a lot of files that have distinct filenames, i don't think that is working properly.
Another example would be with the movie files.
Using your script it names them

000000XX.MOVIE

but with the original it was named like this:

000000XX.mpg

It seems like sometimes your script is using the wrong filetype names when its dumping the contents.

[Shokoniraya]

Code: Select all

#Growlanser .dat script
#just use on raw files
#Script by Shokoniraya

get SIGNATURE long #FLK\x00
get UNKNOW_1 short
get UNKNOW_FLAG short #bit dept for something or a table offset?
get HEADER_FLAG long #it tell us file used as a package or raw size?
get PADDING_NUMBER long
get UNKNOW_2 long
get TOTAL_ARCHIVE_SIZE long #size of current file

get FILES_COUNT long
get UNKNOW_3 long #a flag or maximum header size that game can handle it?


string FILE_EXTENSION = ""
math SLOT_COUNT = 0
for i = 0 < FILES_COUNT
putarray 1 SLOT_COUNT FILE_EXTENSION
math SLOT_COUNT + 1
next i

savepos TABLE_OFFSET
xmath TABLE_SIZE "FILES_COUNT * 8"
goto TABLE_SIZE 0 SEEK_CUR

do
get START_COUNT short
get EXTENSION_RATE short
get NULL long
getdstring FILE_EXTENSION 8

math EXTENSION_COUNT = START_COUNT
for e = 0 < EXTENSION_RATE
putarray 1 EXTENSION_COUNT FILE_EXTENSION
math EXTENSION_COUNT + 1
next e

while FILE_EXTENSION != ""

math NUMBERIC_COUNT = 0
goto TABLE_OFFSET
for i = 0 < FILES_COUNT
string NUMBERIC_COUNT_STRING = ""
string NUMBERIC_COUNT_STRING + NUMBERIC_COUNT
string ZERO_NUMBERIC = "00000000"
strlen NUMBERIC_COUNT_LENGTH NUMBERIC_COUNT_STRING
string ZERO_NUMBERIC - NUMBERIC_COUNT_LENGTH
string ZERO_NUMBERIC + NUMBERIC_COUNT
getarray FILE_EXTENSION 1 NUMBERIC_COUNT

get FILE_OFFSET long #padded to PADDING_NUMBER. if PADDING_NUMBER is 0, then it padded to 2048
if PADDING_NUMBER = 0
math PADDING_NUMBER = 2048
endif

math TEST_OFFSET = FILE_OFFSET
math TEST_OFFSET x PADDING_NUMBER

if TEST_OFFSET = FILE_OFFSET
math FILE_OFFSET x PADDING_NUMBER
endif

get FILE_SIZE long

string FILE_NAME = ""
string FILE_NAME + ZERO_NUMBERIC
string FILE_NAME + "."
string FILE_NAME + FILE_EXTENSION

log FILE_NAME FILE_OFFSET FILE_SIZE


savepos CHECKPOINT
goto 20
get ARCHIVE_SIZE asize
put ARCHIVE_SIZE long
goto CHECKPOINT

math NUMBERIC_COUNT + 1
next i

[Risae]

Holy crap, Shokoniraya, you are so awesome!! Thank you so so much!!
I believe it works right now, nothing seems out of the ordinary compared to the original quickBMS script.

With your script i actually found something cool!
Inside GL6_CHAR.DAT, GL6_FACE.DAT and GL6_MAP.DAT there seem to be files that hold all the file names in it!:

https://imgur.com/a/ypSK9al

Do you think its somehow possible to make the script use of that information to name the files?

[Shokoniraya]

thanks to sir aluigi, i just wrote script but he is the only who that you should say thanks for that for his QuickBMS
i will check it tomrrow and see what can i do about names


and this topic should move to Game Archive

[Shokoniraya]

some of this names is not right, and it's just .def (define)
i suppose this names packed as a log file

[Risae]

Hi Shokoniraya,

thank you very much for your reply!
Ah, ok. It looked to me like its some sort of file list.
Thank you very much for all of your help!