Dishonored 2 shared_2_3

swinei · Post by **swinei** » Fri Jan 13, 2017 8:42 pm

Hey everybody,

dishonored2.bms can currently only extract game#.resources files, not shared_2_3.sharedrsc. I tried adding support for sharedsrc but extraction fails on some compressed data. My modified version:

Code: Select all

# Dishonored 2
# script for QuickBMS http://quickbms.aluigi.org

endian big
get EXT extension
if EXT == "index" || EXT == "resources"

    open FDDE "index"
    open FDDE "resources" 1
    open FDSE "shared_2_3.sharedrsc" 2
    get DUMMY byte  # 0x05 for index and 0x04 for resources
    idstring "SER"
    get SIZE long
    getdstring ZERO 0x18
    get FILES long
    for i = 0 < FILES
        get IDX long
        endian little
        for x = 0 < 3
            get NAMESZ long
            getdstring NAME NAMESZ
        next x
        endian big
        get OFFSET longlong
        get SIZE long
        get ZSIZE long
        get ZERO long
        get FLAGS long
        get ZERO short
        if FLAGS > 0
            if ZSIZE == SIZE
                log NAME OFFSET SIZE 2
            else
                #clog NAME OFFSET ZSIZE SIZE 2 #fails here
            endif
        else
            if ZSIZE == SIZE
                log NAME OFFSET SIZE 1
            else
                clog NAME OFFSET ZSIZE SIZE 1
            endif
        endif
    next i

else

    comtype zlib_noerror
    get SIZE asize
    get NAME basename
    string NAME + "_unpack."
    string NAME + EXT
    clog NAME 0 SIZE SIZE

endif

This is the error I get when trying to extract game1:

Code: Select all

Error: the compressed zlib/deflate input is wrong or incomplete (-3)
Info:  algorithm   1
       offset      0000000028f03182
       input size  0x0000000000002b46 11078
       output size 0x0000000000040014 262164
       result      0xffffffffffffffff -1

Error: the uncompressed data (-1) is bigger than the allocated buffer (15729302)

This is a 010 template to look at the .index files:

Code: Select all

//------------------------------------------------
//--- 010 Editor v7.0.2 Binary Template
//
//      File: 
//   Authors: 
//   Version: 
//   Purpose: 
//  Category: 
// File Mask: 
//  ID Bytes: 
//   History: 
//------------------------------------------------
BigEndian();
local uint i;
local uint j;
local int files_var;

struct HEADER {
    byte    type[1] <format=decimal>;
    char    idstring[3];
    int     size;
    char    zero[24];
    int     files;
    files_var = files;
} header <bgcolor=cLtGray>;

for(i=0;i<files_var;i++) {
    struct DATA {
        BigEndian();
        int     idx;
    
        LittleEndian();
        

        local string custom_read;
        for(j=0;j<3;j++) {
    
            struct FILES {
        
                int     namesz;
                char    name[namesz];
                if(j==1) {
                    if(namesz) {
                        custom_read = name;
                    } else {
                        custom_read = "n/a";
                    }
                }
            } record <read=read_FILES>;
            
    
        }
    
        BigEndian();
        int64   offset;
        int     size;
        int     zsize;
        int     zero;
        int     flags;

        LittleEndian();
        int16   flags2;
        
    } record <read=read_DATA>;
}

string read_FILES(FILES &in) {
    return in.name;
}

string read_DATA(DATA &in) {
    local string out;
    SPrintf(out, "%d | %u | %s", in.flags, (in.size - in.zsize), in.custom_read);
    return out;
}

I'd like to extract all available files. Can somebody help?

aluigi · Post by **aluigi** » Sat Jan 14, 2017 2:13 pm

What you uploaded are just useless huge data files.
All the necessary content for the script is in the index files, you must upload them.

swinei · Post by **swinei** » Sat Jan 14, 2017 3:54 pm

The index files are included in the archives. Here is index files only:

index.7z

aluigi · Post by **aluigi** » Sat Jan 14, 2017 5:12 pm

There was no index in the shared_2_3 I downloaded and there is no index for that archive in the new zip too.

swinei · Post by **swinei** » Sat Jan 14, 2017 6:27 pm

Yeah, there's only game#.index files, if FLAGS is set then the data isn't in game#.reources but in shared_2_3.sharedrsc

Code: Select all

    open FDSE "shared_2_3.sharedrsc" 2
    [...]
        if FLAGS > 0
            if ZSIZE == SIZE
                log NAME OFFSET SIZE 2
            else
                #clog NAME OFFSET ZSIZE SIZE 2 #fails here
            endif
        else

The modified dishonored2.bms I posted can extract all uncompressed files but fails on compressed ones.
You could try it with game1. Download the game1 archive and put the files in the same folder as shared_2_3.sharedrsc, then try to extract game1.index.

aluigi · Post by **aluigi** » Sat Jan 14, 2017 7:59 pm

Ok now I understand what you mean

As far as I can see the FLAGS set to 32 is the only different thing (for example the two ZERO fields are still zero) but the offset doesn't match at all the data in the shared archive, it can be clearly verified with offzip.
Even using the data from shared sequentially (for example the first file with FLAGS 32 in game1.index is the first file at offset 4 of shared) fails and doesn't match the data.
Basically there are no information that say how to get the data from the shared archive.

swinei · Post by **swinei** » Sat Jan 14, 2017 8:47 pm

Bummer! Thanks for checking it out

So now all that I can try is the compression type scanner, right? The correctly uncompressed file should have the SIZE from index if I understand correctly?

aluigi · Post by **aluigi** » Sun Jan 15, 2017 6:27 am

offzip -a will be able to find and decompress the compressed files only, so all the others (many) will be invisible.
In these cases there is just no solution.

michalss · Post by **michalss** » Sun Jan 15, 2017 9:58 pm

shared_2_3 is only addtiional extra archive it has not own index but it is shared between all other archives. On struct you can see index file as flag..

swinei · Post by **swinei** » Tue Jan 17, 2017 1:28 am

I tried comtype_scanner on a file I cut out of shared_2_3 and could not get a valid file from it, I checked all of the resulting files.

I tried offzip -a, I got a lot of "zlib Z_DATA_ERROR, the data in the file is not in zip format or uses a different windowBits value (-z). Try to use -z -15" but my HD space ran out anyway. I'll try offzip again once I make some room.

The attachment is a list of all compressed files in shared_2_3.

Okay so if all of this doesn't work the last hope is to step through the loading of the archives during game launch I think and I have no clue about that.

Any other ideas? I really want those bimages

aluigi · Post by **aluigi** » Tue Jan 17, 2017 8:04 am

Do NEVER use comtype_scanner if you don't know what it does. I repeat, NEVER.

I already said that offzip does the job for the compressed data (zlib).

volfin · Post by **volfin** » Tue Jan 17, 2017 8:19 am

I found many instances where the zlib data decompresses fine. It's simply that there's deleted entries present in the records. Luckily there's a way to tell which entries are valid.

This script works for me on game2.index and game3.index with shared_2_3.sharedrsc
( of course all the entries in game1.index are invalid, because it would be called shared_1_2_3.sharedrsc if they weren't :lol:

):

Code: Select all

# Dishonored 2
# script for QuickBMS http://quickbms.aluigi.org

endian big
get EXT extension
if EXT == "index" || EXT == "resources"

    open FDDE "index"
    open FDDE "resources" 1
    open FDSE "shared_2_3.sharedrsc" 2
    get DUMMY byte  # 0x05 for index and 0x04 for resources
    idstring "SER"
    get SIZE long
    getdstring ZERO 0x18
    get FILES long
    for i = 0 < FILES
        get IDX long
        endian little
        for x = 0 < 3
            get NAMESZ long
            getdstring NAME NAMESZ
        next x
        endian big
        get OFFSET longlong
        get SIZE long
        get ZSIZE long
        get ZERO long
        get FLAGS long
        get VALID short
        if FLAGS > 0
            if VALID == 32768 
               if ZSIZE == SIZE
                   log NAME OFFSET SIZE 2
               else
                   clog NAME OFFSET ZSIZE SIZE 2
               endif
         endif
        else
            if ZSIZE == SIZE
                log NAME OFFSET SIZE 1
            else
                clog NAME OFFSET ZSIZE SIZE 1
            endif
        endif
    next i

else

    comtype zlib_noerror
    get SIZE asize
    get NAME basename
    string NAME + "_unpack."
    string NAME + EXT
    clog NAME 0 SIZE SIZE

endif

Good work figuring this out (almost). I thought we'd never get these files out.

Some of the mips seem to duplicate already existing mip files, but at higher resolution. so it's best to choose to overwrite, so you get the higher rez textures.

aluigi · Post by **aluigi** » Tue Jan 17, 2017 8:29 am

volfin wrote:This script works for me on game2.index and game3.index with shared_2_3.sharedrsc
( of course all the entries in game1.index are invalid, because it would be called shared_1_2_3.sharedrsc if they weren't ):

That explains why everything failed on game1

I have only one doubt, why there are flags set to 32 on game1.index if there is no shared_*1* available?

aluigi · Post by **aluigi** » Tue Jan 17, 2017 8:50 am

Script 0.2.

aluigi · Post by **aluigi** » Tue Jan 17, 2017 9:19 am

Another fix of the script, unfortunately that VALID 16bit field is not clear because everything fails if it's different than 0x8000
Currently I opt for using the shared file only if the it's 0x8000 and going on the normal archive if it has other values, it may result on errors.

volfin · Post by **volfin** » Wed Jan 18, 2017 5:49 am

aluigi wrote:Another fix of the script, unfortunately that VALID 16bit field is not clear because everything fails if it's different than 0x8000
Currently I opt for using the shared file only if the it's 0x8000 and going on the normal archive if it has other values, it may result on errors.

You're probably right. It's hard to really say what they are doing there. But we do the best we can

swinei · Post by **swinei** » Wed Jan 18, 2017 7:05 pm

Awesome! I ran dishonored2.bms 0.2.1 on all the index files and it worked without any errors.

Thank you all!

swinei · Post by **swinei** » Wed Jan 18, 2017 7:08 pm

aluigi wrote:Do NEVER use comtype_scanner if you don't know what it does. I repeat, NEVER.

I already said that offzip does the job for the compressed data (zlib).

I think I know what it does, it tries all possible ways to decompress a file, right? Is there something that can go really wrong when using it?

aluigi · Post by **aluigi** » Wed Jan 18, 2017 7:17 pm

Basically it tries all the over 600 decompression algorithms embedded in quickbms on the whole input file, so such file must contain the whole compressed data from offset 0 till its end.

The reason why your usage was incorrect in this context is that we already know the compression algorithm (zlib) so there was no reason to try it.

Anyway, yeah there are no side effects except wasting a couple of minutes and few megabytes on disk/ramdisk

Different story if you try it on a file of many megabytes (like some users did)... prepare the pop corn

swinei · Post by **swinei** » Wed Jan 18, 2017 7:29 pm

Ah I see

So the compression algorithm was found with offzip, but how did you know that it was actually working and not giving false positives?

ZenHAX

Dishonored 2 shared_2_3

Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3

Re: Dishonored 2 shared_2_3