Deus Ex: Human Revolution Director's Cut (Wii U) - bigfile/dlcpack files

[AnonBaiter]

Here's the sample.

[aluigi]

http://aluigi.org/bms/deus_ex_human_revolution.bms

[AnonBaiter]

Oh look, I got an error:

Code: Select all

Error: incomplete input file 0: E:\DeusEx\DeusEx\data\content\bigfile-wiiu.000
       Can't read 64 bytes from offset 7f1a7000.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0     0%   443734     522956800

Last script line before the error or that produced the error:
  18  log "" OFFSET SIZE

Is this a common thing on the files themselves or just an error?

[aluigi]

The problem is not the script but the information reported by the archive.
The archive has a list of offsets that must be multiplied by 0x800 and most of them are outside the size of the archive.
If you use quickbms -0 or -l, and then use sort on its output, you can see that the initial offsets are all correct.

Now I have updated the script to 0.1.1 for checking if the offset is bigger than the size of the archive, I highly suggest to use quickbms_4gb_files.exe to allow the check to work correctly.

[AnonBaiter]

Well I think you're right.
Thanks for the update but what about the following files like .001, .002, and so on? I was only able to extract the .000 file and nothing else. At least it did check on something.

[AnonBaiter]

Okay, how about this:
Is there any way to extract the files in a sequential order like .000, .001, .002 and up until .027 for the "bigfile-wiiu" archive? The same can also be applied to the "dlcpack-wiiu" archive, which ends at the .005 file.

[aluigi]

Aaaaaahhhh... now I understand what's the trick.
I have updated the script to version 0.2 but it's not easy to test it here.
Can you try it and give me a feedback?
Basically if all the files have been extracted and if their content is correct.

[AnonBaiter]

This is what I got, even on the quickbms_4gb_files.exe executable:

Code: Select all

  offset           filesize   filename
--------------------------------------
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   2218534    522956800
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.000
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          522956800
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.001
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          521560064
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.002
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          518082560
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.003
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          522348544
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.004
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          428470272
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.005
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          496232448
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.006
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          523843584
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.007
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          510908416
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.008
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524265472
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.009
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          522264576
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.010
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524253184
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.011
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524283904
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.012
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524277760
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.013
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          346978304
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.014
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524285952
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.015
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524277760
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.016
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524242944
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.017
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524279808
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.018
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          464631808
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.019
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          503169024
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.020
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.021
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.022
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.023
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.024
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.025
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.026
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524150784
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.027
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          99993600
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.028

- 0 files found in 5 seconds

It didn't extract anything, it only displayed all the archive files instead.

[aluigi]

That one seems the part of the archives scanner, the extraction part is missing...
Do you have the same error also with dlcpack-wiiu.000?

[AnonBaiter]

^ Same error with that "dlcpack-wiiu" file too, so the error is confirmed.

[aluigi]

Can you try version 0.2a?

[AnonBaiter]

This is what I got out of your "0.2a" version of the script:

Code: Select all

  offset           filesize   filename
--------------------------------------
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   2218534    522956800
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.000
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          522956800
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.001
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          521560064
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.002
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          518082560
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.003
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          522348544
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.004
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          428470272
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.005
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          496232448
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.006
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          523843584
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.007
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          510908416
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.008
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524265472
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.009
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          522264576
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.010
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524253184
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.011
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524283904
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.012
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524277760
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.013
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          346978304
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.014
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524285952
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.015
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524277760
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.016
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524242944
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.017
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524279808
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.018
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          464631808
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.019
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          503169024
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.020
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.021
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.022
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.023
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.024
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.025
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524288000
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.026
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524150784
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.027
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          99993600
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.028
- enter in folder E:\DeusEx\DeusEx\data\content
- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.012
  000000017f1a7000 938164     0000000000000000.dat

Error: incomplete input file 0: E:\DeusEx\DeusEx\data\content\bigfile-wiiu.012
       Can't read 64 bytes from offset 000000017f1a7000.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0     0%   0          524277760

Last script line before the error or that produced the error:
  68  log "" OFFSET SIZE

Same error with dlcpack-wiiu:

Code: Select all

  offset           filesize   filename
--------------------------------------
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   750134     524261376
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.000
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524261376
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.001
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524265472
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.002
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524163072
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.003
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          524281856
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.004
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   0          346869760
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.005
- enter in folder E:\DeusEx\DeusEx\data\content
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.000
  0000000003309800 859844     0000000000000000.dat
- enter in folder E:\DeusEx\DeusEx\data\content
  coverage file 0     0%   859908     524261376
- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.002
  000000004524d000 75360      0000000000000001.dat

Error: incomplete input file 0: E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.002
       Can't read 64 bytes from offset 000000004524d000.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0     0%   0          524163072

Last script line before the error or that produced the error:
  68  log "" OFFSET SIZE

E:\quickbms>

[aluigi]

Very good: 0.2.1

[AnonBaiter]

Okay, so with your new version of the script the extraction started to go well until a certain point:

Code: Select all

- open input file E:\DeusEx\DeusEx\data\content\bigfile-wiiu.015
  000000001f3fb000 48528      0000000000003140.dat

Error: incomplete input file 0: E:\DeusEx\DeusEx\data\content\bigfile-wiiu.015
       Can't read 38288 bytes from offset 000000001f3fd800.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0     0%   10304      524277760

Last script line before the error or that produced the error:
  69  log "" OFFSET SIZE

This is also the same for the "dlcpack-wiiu" files:

Code: Select all

- open input file E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.001
  000000001f3f9800 28828      000000000000185e.dat

Error: incomplete input file 0: E:\DeusEx\DeusEx\data\content\dlcpack-wiiu.001
       Can't read 24732 bytes from offset 000000001f3fa800.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0     0%   4160       524265472

Last script line before the error or that produced the error:
  69  log "" OFFSET SIZE

[aluigi]

So basically it means that the files can be even splitted in two sequential archives I guess. Crazy.
The alternative solution is merging all the files creating a huge unique file but it's boring, so I have updated the script 0.2.2 for handling these splitted files.

[AnonBaiter]

Same error:

bigfile-wiiu

Code: Select all

Error: incomplete input file 0: E:\DeusEx\DeusEx\data\content\bigfile-wiiu.015
       Can't read 38288 bytes from offset 000000001f3fd800.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0     0%   10240      524277760

Last script line before the error or that produced the error:
  79  log MEMORY_FILE OFFSET SIZE

dlcpack-wiiu

Code: Select all

to be reported

So literally, unless you decide to finally fix it, that leaves me with no other options: to merge all these files into one.
You might as well make a script to merge them anyway, but I doubt that will work.

[aluigi]

0.2.3, hope this is the good one

[AnonBaiter]

Got it! I'll report the results when it's done.

Meanwhile, I've discovered something interesting. Out of the files contained in both of these attachments, only 4 of them have actual filename information. They were located right at the end of the splitted files.

[AnonBaiter]

Okay now speaking of extracting the "bigfile-wiiu" file, I did it... at least until it stopped at "000000000000e351.dat" file, which gave me this error:

Code: Select all

Error: incomplete input file 0: E:\DeusEx\DeusEx\data\content\bigfile-wiiu.023
       Can't read 564069511 bytes from offset 000000001f400000.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file 0   100%   524288000  524288000

Last script line before the error or that produced the error:
  87  log MEMORY_FILE 0 SIZE

I couldn't capture the exact moment this error happened since I messed up with the cmd, thus I had to use the -l option to reproduce it. Should be enough proof now.

[aluigi]

I have updated the script to 0.2.3a for adding a sort of "break" when the correct archive is found but I don't know if it's the solution.