Mario + Rabbids .sdfdata, .sdftoc

Extraction and unpacking of game archives and compression, encryption, obfuscation, decoding of unknown files
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Okay so first things first, yes i tried the bms script made for the Division and yes it mostly works but there are issues.

Basically, it eventually runs into an error like this:

Code: Select all

- open input file C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-C-2265.sdfdata
  00000000 119        moria/baked/art/[assets]/[generictexture]/gn_roof_tile_bitumen_01_d.dds

- The following output file already exists:
  moria\baked\art\[assets]\[generictexture]\gn_roof_tile_bitumen_01_d.dds
  Do you want to overwrite it?
    y = overwrite (you can use also the 'o' key)
    n = skip (default, just press ENTER)
    a = overwrite all the files without asking
    r = automatically rename the files with the same name
    s = skip all the existent files without asking



Some files are split across the SDFDATA archives and the script runs into issues with this, but
no matter what option you choose here it immediately goes into this:

Code: Select all

Error: incomplete input file 1: C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-C-2265.sdfdata
       Can't read 114 bytes from offset 00000005.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 1   100%   5          5          . offset 00000005

Last script line before the error or that produced the error:
  127 log name packageOffset decompressedSize 1


The script also doesn't support the merging of DDS headers from the SDFTOC file to the output DDS files in the extracted SDFDATA, so it has to be done manually at the moment, Although theres a whole other issue with those that i'd need to go post about on the Graphics Files board instead(Basically its an obscure DXGI format that uses ASTC i think)

Running the script on smaller amounts of SDFDATA at a time seems to help with the issues a bit but the file splitting is still an issue.
One of my friends was trying to modify the tool the script is based off (Rogue_SDF) to work with this game but ran into issues with "chunks"? That's all i know.
I'll send samples if required but i think a full dump may be needed to properly investigate the issues.

Any help would be appreciated :)
In the meantime here is a screenshot of one of the models that managed to extract properly and load in Noesis.
Image
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by aluigi »

Do you mean you have already tried the http://aluigi.org/bms/ubisoft_sdf.bms script?
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Yes, that's the one.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by aluigi »

Could you please upload the sdftoc file?
Basically sdftoc is telling you that sdf-C-2265.sdfdata contains a file with a duplicated name and a size of 114 bytes while that "container" is 5 bytes only, so there is something weird or something else to implement.
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by aluigi »

Long story short, I don't know what's the problem.

Everything is correct till the code "for chunkIndex = 0 < count1" reads ch3 from offset 0x0001a560 of MEMORY_FILE which is 0x00, and this is the only ch3 with this value.
So what I did was ignoring the instructions if ch3 is 0x00 (which is an invalid value indeed) and apparently everything works correctly :)
Script 0.1.3a, let me know if you get other errors.
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Same errors as before but it happens slightly later i think.
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Okay, i forgot about another error that happens if it doesn't run into any existing files. .

Code: Select all

Error: incomplete input file -1:
       Can't read 3 bytes from offset 00527172.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file -1  117%   6355722    5402994    . offset 00527172

Last script line before the error or that produced the error:
  87  get fileId long MEMORY_FILE
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by aluigi »

Yesterday I updated the script for supporting version 0x17 and it may fix that error.
Anyway don't worry since it just means you have extracted everything and you can ignore it.
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

The issue with files split across archives is still there :(
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

The other issue is gone, forgot to mention that somehow.
.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by aluigi »

Can you remind me what's the issue with "files split across archives"?
The script supports the various sdfdata and I'm not aware of files that are located half in one and half in another archive.
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Well one would be Mario's model, i think.

Code: Select all

- enter in folder C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data
- open input file C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-A-0208.sdfdata
  00000000 0          moria/baked/art/animation/01-chr/02-her/mario/chr_her_mario_tpose_01.mmb
+ 0001b283 65536      moria/baked/art/animation/01-chr/02-her/mario/chr_her_mario_tpose_01.mmb
+ 00021b5c 65536      moria/baked/art/animation/01-chr/02-her/mario/chr_her_mario_tpose_01.mmb
+ 000294fa 30365      moria/baked/art/animation/01-chr/02-her/mario/chr_her_mario_tpose_01.mmb
- enter in folder C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data
  coverage file 1     0%   72403      24936897   . offset 00000000
- open input file C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-A-0209.sdfdata
  000092f9 2230304    moria/baked/art/animation/01-chr/02-her/mario/chr_her_mario_tpose_01.mmb

- The following output file already exists:
  moria\baked\art\animation\01-chr\02-her\mario\chr_her_mario_tpose_01.mmb
  Do you want to overwrite it?
    y = overwrite (you can use also the 'o' key)
    n = skip (default, just press ENTER)
    a = overwrite all the files without asking
    r = automatically rename the files with the same name
    s = skip all the existent files without asking



And no matter what you choose this happens:

Code: Select all

Error: incomplete input file 1: C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-D-3846.sdfdata
       Can't read 10341 bytes from offset 00000005.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 1   100%   5          5          . offset 00000005

Last script line before the error or that produced the error:
  155 clog name packageOffset compSize CHUNK_SIZE 1

Press ENTER or close the window to quit
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

SDFDATA 3846 seems to be a 5 byte dummy file that apparently contains 2 parts of a model???
Something is really odd here.

Code: Select all

- open input file C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-D-3846.sdfdata
  00000000 0          moria/baked/art/animation/01-chr/03-enm/globetrotter/chr_enm_shd_globetrotter_tpose_01.mmb
+ 00000000 65536      moria/baked/art/animation/01-chr/03-enm/globetrotter/chr_enm_shd_globetrotter_tpose_01.mmb


Image
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Removing all the dummy files still ends up with a similar issue,

Code: Select all

- open input file C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-B-1702.sdfdata
  00c79b12 54030      moria/baked/art/animation/01-chr/03-enm/shielder/chr_enm_lvp_shielder_tpose_01.mmb
- enter in folder C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data
  coverage file 1     0%   24139      25137233   . offset 00000000
- open input file C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-B-1703.sdfdata
  000081da 12257012   moria/baked/art/animation/01-chr/03-enm/shielder/chr_enm_lvp_shielder_tpose_01.mmb

Error: incomplete input file 1: C:\Users\faint\Documents\SwitchDumps\MarioRabbidsKingdomBattle\romfs\moria\sdf\nx\data\sdf-B-1703.sdfdata
       Can't read 101529 bytes from offset 008ef541.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 1    99%   9335655    9368897    . offset 008ef541

Last script line before the error or that produced the error:
  140 log name packageOffset decompressedSize 1

Press ENTER or close the window to quit
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by aluigi »

I already provided a full explanation of why it's impossible to extract N bytes from a dummy archive containing only 5 bytes.
aluigi
Site Admin
Posts: 12984
Joined: Wed Jul 30, 2014 9:32 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by aluigi »

Let's say that ch3 equal than zero should be handled as a "break" instead of a "continue" (like it's right now).
I have updated the script for trying this other method and fixing an unrelated bug in DUMP_FILE.
Let me know if the situation is now better.
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Still getting the
CHUNK_SIZE 1
error from it trying to extract from a dummy file, so no change from the previous version basically.

Very confusing issue...
I'll upload a batch of samples shortly. (i already have the entire game uploaded to google drive so if the samples aren't enough help i can pm a link to that or something, don't wanna link something like that publicly)
CosmicDreams
Posts: 100
Joined: Thu Sep 13, 2018 6:38 pm

Re: Mario + Rabbids .sdfdata, .sdftoc

Post by CosmicDreams »

Alright here's the samples.
There was a file in a folder called chunkmapping.mchunks at the
romfs\moria\sdfconfig\nx
directory that i thought i should include.
Everything else is from the
romfs\moria\sdf\nx\data
directory, the sdfver file is probably useless but i included it anyway.
https://drive.google.com/file/d/1HSKLQl ... sp=sharing

EDIT: Of course i forgot the link.