MACROSS_PSP

Extraction and unpacking of game archives and compression, encryption, obfuscation, decoding of unknown files
Chthonic
Posts: 12
Joined: Sat Mar 05, 2016 3:57 pm

MACROSS_PSP

Post by Chthonic »

I was hoping someone can help me with a Game ARCHIVE Extraction for a PSP and PS3 game. I checked the coding and it's PIDX0 which I haven't seen before

Here's the Link to the PSP game Archive - Macross Ultimate Frontier
https://www.dropbox.com/s/ik4nkw08ny3w6 ... s.rar?dl=0

Here's the Link to The PS3 Game Archive- Macross 30
https://www.dropbox.com/s/m0o7ns8uqogwc ... 2.rar?dl=0
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

Chthonic
Posts: 12
Joined: Sat Mar 05, 2016 3:57 pm

Re: MACROSS_PSP

Post by Chthonic »

I really appreciate your help aluigi. You're Awesome. I attempted to run the new script and I'm running into an error

78 clog MEMORY_FILE OFFSET ZSIZE SIZE
Error: the uncompressed date (-1) is bigger than the allocated buffer (65536)
Error: The compressed zlib/deflate input is wrong or incomplete (-3)
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

I used the script with both the files you provided but there are no errors.
If it's a different file then you have to upload it.
Chthonic
Posts: 12
Joined: Sat Mar 05, 2016 3:57 pm

Re: MACROSS_PSP

Post by Chthonic »

Here's the other Data file for Macross 30
https://www.dropbox.com/s/yoqye6xqwbxwu ... a.rar?dl=0

The other is a PSP game Macross Triangle Frontier
https://www.dropbox.com/s/xvw2phtpmpi01 ... F.rar?dl=0

The both use the PIDX0
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

Script 0.1.1.
The GIM files that will be placed in the "unknown" folder starting with the magics " 3;0" or " 3;1" can't be decompressed, they are probably obfuscated or use some unknown string compression algorithm.
Chthonic
Posts: 12
Joined: Sat Mar 05, 2016 3:57 pm

Re: MACROSS_PSP

Post by Chthonic »

Is there a new link for the Script 0.1.1?
Chthonic
Posts: 12
Joined: Sat Mar 05, 2016 3:57 pm

Re: MACROSS_PSP

Post by Chthonic »

My apologies I saw the update-
steinrock
Posts: 1
Joined: Fri Dec 18, 2020 7:10 pm

Re: MACROSS_PSP

Post by steinrock »

Hello there.
Sorry for necropost.
I did a small investigation and found that bms script working normally only with Macross Ultimate Frontier macross.dat. With Macross Triangle Frontier macross.dat it's working partially (i have a suspicion that script does not extract all files). With PS3 versions of macross that script is not working. And I found that macross.dat only contains resourses like icons, voices, backgrounds and load screens, not text files. And this script not working with fileset.dat, this file use another structure. In MTF this script extract from fileset.dat some files seems backgrounds and maps but no more.
P.S. Files on MUF macross.dat is not encrypted
I hope this small information may be useful
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

Feel free to provide some sample files.
gmproxy
Posts: 1
Joined: Wed Aug 18, 2021 7:00 pm

Re: MACROSS_PSP

Post by gmproxy »

Greetings. I am currently trying to work on Macross 30 too, and this is what it said after trying that pidx script:

Image

Couldn't find more info about it, sorry.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

We still need sample files understanding what's wrong.

Considering that the files of this topic were DAT (like "macross.dat"), I suppose yours is a completely different format, for example some pack.idx files can be parsed by this script http://aluigi.org/bms/dragon_ball_z_boz.bms (I doubt your file is this one)
alanm
Posts: 21
Joined: Mon Aug 17, 2020 4:54 am

Re: MACROSS_PSP

Post by alanm »

aluigi wrote:We still need sample files understanding what's wrong.

Considering that the files of this topic were DAT (like "macross.dat"), I suppose yours is a completely different format, for example some pack.idx files can be parsed by this script http://aluigi.org/bms/dragon_ball_z_boz.bms (I doubt your file is this one)


Here are the sample files for Macross 30: pack.idx and fileset0.dat.
https://www.mediafire.com/file/a6rkr7gxukgcdr3/pack.7z/file
Thank you for providing the macross_pidx.bms script which works well for the data.dat archive.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

So, as far as I understand, there is a pack.idx and a pack.dat, correct?
Because pack.idx doesn't contain any data, it's only the table.

fileset0.dat has a bit different format, still PIDX but the struct is different (names offsets are in sequence, table is not the same struct).
alanm
Posts: 21
Joined: Mon Aug 17, 2020 4:54 am

Re: MACROSS_PSP

Post by alanm »

There are only a few files in game disc pack directory as below. Does not seems to have a pack.dat file.
Image
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

I guess I finally figured everything :D
Script 0.2
http://aluigi.org/bms/macross_pidx.bms

The script also works if you select pack.idx and one or more dat files are missing, in which case it will just continue.
alanm
Posts: 21
Joined: Mon Aug 17, 2020 4:54 am

Re: MACROSS_PSP

Post by alanm »

Really appreciate your help. So it seems both pack.idx and data.dat have overlapping offset table entries point to the same set of files/packs. I have both data[2].dat and fileset[0,1,2].dat in the same directory. I am curious that the log messages below does not mention fileset(n).dat, although fileset.dat path was mentioned inside pack.idx. I got the dialog text files I want from data.dat so I am good. thank you.

quickbms.exe -l backup\macross_pidx.bms pack.idx > pack.lst

QuickBMS generic files extractor and reimporter 0.11.0
by Luigi Auriemma
e-mail: me@aluigi.org
web: aluigi.org
(Apr 5 2021 - 13:56:34)

quickbms.com Homepage
zenhax.com ZenHAX Forum
@zenhax @quickbms Twitter & Scripts

- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\pack.idx
- open script backup\macross_pidx.bms

offset filesize filename
--------------------------------------
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\pack.idx
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 0 2695864 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 4 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 3% 35793509 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 0 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 5933555 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 6534537 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 2555777 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 74% 1042045302 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 7183948 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 1% 16977819 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 81% 772772461 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/lua.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\lua.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 988 786432 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 1% 27193080 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/lua.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\lua.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 228 786432 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 4580801 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/sound.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\sound.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 1% 4141984 371838320 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 0 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/sound.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\sound.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 0 371838320 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 52 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/init.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\init.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 12 8258864 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data2.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 2% 26721512 950824960 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
coverage file 0 0% 32 1395880848 . offset 00000000
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\data/pack/shaders.dat
- enter in folder Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack
- open input file Z:\dev_hdd0\disc\BLJS10184\PS3_GAME\USRDIR\data\pack\shaders.dat

- 59906 files found in 1049 seconds
coverage file 0 0% 105420 87894096 . offset 00000000
coverage file 1 97% 2638708 2695864 . offset 00160f20
coverage file -1 0% 0 174904 . offset 0002ab38
coverage file -2 100%! 1317702 1193340 . offset 0011d5a4
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: MACROSS_PSP

Post by aluigi »

Yeah, it's the same thing I noticed (but I forgot to say).

data\pack\fileset0.dat is available in the list of pack names but is never used, there are no files referenced to it.
Indeed even fileset0.dat itself has 0 files reported:

Code: Select all

00000000 idstr              "PIDX" 4
  50 49 44 58                                       PIDX
00000004 get     PACKS_OFF  0x00000030 4
00000008 get     PACKS      0x00000001 4
0000000c get     INFO_OFF   0x00000050 4
00000010 get     FILES      0x00000000 4
00000014 get     DUMMY      0x00000000 4
00000018 get     OFFSET2    0x00000050 4
0000001c get     OFFSET2_SIZE 0x00000dfc 4
00000020 get     NAMES_OFF  0x00000e4c 4
00000024 get     NAMES_SIZE 0x00000635 4

And there are no files referenced even if you manually set FILES to a greater number or force the table offset to a different position (like 0x2ac)... yeah everything is correct.

So I investigated a bit on it and it looks like it's a sort of slightly different format, FILES is zero but a new FILES field is available later.
I figured out the differences and updated the macross_pidx.bms script to version 0.2.1 for supporting it (obviously pack.idx will still ignore it).

The files it dumps have an FSTS format (even if all the extensions are differents) that is another archive format with segs-compressed files.
I made a script for these files too:
http://aluigi.org/bms/macross_fsts.bms

I hope it's now really 100% covered :)
alanm
Posts: 21
Joined: Mon Aug 17, 2020 4:54 am

Re: MACROSS_PSP

Post by alanm »

Thank you for getting to the bottom of it. It make sense and I think you have all the archives covered :D .
alanm
Posts: 21
Joined: Mon Aug 17, 2020 4:54 am

Re: MACROSS_PSP

Post by alanm »

Hi, I was looking for gop_scenariotext.gop file which macross_pidx.bms can extracted from data.dat. I changed some Japanese text to English and re-imported it back to data.dat but game did not show my changes. It turns out data.dat has a second file table in offset2. These files are non-compress raw FSTS. I found another copy of gop_scenariotext packed in one of these FSTS file. Don't know why this game pack copies of the same file in serval packs. So I make a copy of macroos_pidx.bms and change it to only export the raw fsts file from offset2. Once i got the fsts, I use macross_fsts.bms to extract the text file. reverse this process to reimport the changes. Now I can see the modified text in game. Here is the pidx offset2 script:

https://www.mediafire.com/file/3tc8z2ut6xa3v9y/macross_pidx_offset2.bms/file