Having issues deciphering Evo Engine variant

[sephirothxi]

Been banging my head in frustration for almost a week now trying to figure out how to extract files from this format; I hope someone here can at least point me in the right direction.

These are standard .toc/.cache files from Digital Extremes but they've updated the format enough that the EvolutionUnpacker tool can't pull files correctly. I've tried using various QuickBMS scripts designed for the engine, but consistently run into algorithm problems.

If I'm using signsrch correctly, the files now use comtype dmc2 but I've hit a dead end — either "the data is bigger than the allocated buffer" or "error! out of data".

Sample files: B.Font.cache • B.Font.toc

For convenience, here's the regular Evo Engine BMS script, which throws a "The uncompressed data (-1) is bigger than the allocated buffer (1430)" error:

Code: Select all

# Darkness 2 - Evolution Engine (script 0.2)
#   derived from GMMan's Evolution Engine Extractor
# script for QuickBMS http://quickbms.aluigi.org

quickbmsver "0.5.25a"
comtype evolution

open FDDE "toc"
open FDDE "cache" 1

idstring "\x4e\xc6\x67\x18"
get VER long
get FILES asize
math FILES -= 8
math FILES /= 0x60

math nextDirIndex = 0
putarray 6 nextDirIndex ""
math nextDirIndex += 1
math ENTRIES = 0

# pre-allocate arrays
for i = 0 <= 6
    putarray i FILES ""
next i

for i = 0 < FILES
    get OFFSET longlong
    get TSTAMP longlong
    get ZSIZE long
    get SIZE long
    get RESERVED long
    get ID long
    getdstring NAME 0x40

    if OFFSET == 0xffffffffffffffff
        getarray PATH 6 ID
        string PATH += /
        string PATH += NAME
        putarray 6 nextDirIndex PATH
        math nextDirIndex += 1
    else
        putarray 1 ENTRIES NAME
        putarray 2 ENTRIES OFFSET
        putarray 3 ENTRIES ZSIZE
        putarray 4 ENTRIES SIZE
        putarray 5 ENTRIES ID
        math ENTRIES += 1
    endif
next i

for i = 0 < ENTRIES
    getarray NAME   1 i
    getarray OFFSET 2 i
    getarray ZSIZE  3 i
    getarray SIZE   4 i
    getarray ID     5 i

    getarray FNAME 6 ID
    string FNAME += /
    string FNAME += NAME

    if SIZE == ZSIZE
        log FNAME OFFSET SIZE 1
    else
        endian big
        putvarchr MEMORY_FILE SIZE 0
        log MEMORY_FILE 0 0
        append
        math LIMIT = OFFSET
        math LIMIT += ZSIZE
        for OFFSET = OFFSET < LIMIT
            goto OFFSET 1
            get CHUNK_ZSIZE short 1
            get CHUNK_SIZE short 1
            savepos OFFSET 1
            if CHUNK_ZSIZE == CHUNK_SIZE
                log MEMORY_FILE OFFSET CHUNK_SIZE 1
            else
                clog MEMORY_FILE OFFSET CHUNK_ZSIZE CHUNK_SIZE 1
            endif
            math OFFSET += CHUNK_ZSIZE
        next
        append
        log FNAME 0 SIZE MEMORY_FILE
    endif
next i

[sephirothxi]

Here's a folder->file structure from the file I'm trying to decode:

Code: Select all

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ....Icons....................................................................¤³ÌìÈÒ.è...–...........SaveVisibleSmall.ico

And here's Warframe's equivalent:

Code: Select all

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ....Player..........................................................”\x.....°D.£êÄÒ.Î...(...........FsmTennoKnockdown.animfsm

Is that little extra 945C7801 (”\x) throwing this script off?

I'm using comtype dmc2 and it successfully extracts files to a point, before throwing an "Error: incomplete input file -1" at me. Any advice?

[sephirothxi]

Guess you guys are stumped, huh? Even aluigi!

I've really been trying to figure this out as I'm contributing towards a community site for the game, and having the assets would make things a thousand times easier. Don't really want to be stuck with half-obscured screenshotted images

I'll keep hacking away and see if I have any epiphanies. If I make progress, I'll share it here.

[aluigi]

The 0x8c byte of the compressed data made me think to oodle and... I was right

comtype evolution -> comtype oodle

Now would be cool to find a way to guess when using oodle or evolution, I have just updated the script (0.2.1) to check if VER is >= 0x14

[sephirothxi]

I've tried the updated script, still no luck though — still getting "incomplete input file -1" errors.

I'll attach another toc/cache pair to see if it helps. I'll drop the game's oodle dll in there too, just in case it's different from the one quickbms uses, as I don't think you can call dlls externally:
https://ufile.io/ck518

[aluigi]

No, here everything works perfectly.
I guess you are using an old version of quickbms

[sephirothxi]

I double checked and even redownloaded quickbms to a new folder to be sure, definitely using 0.8.0 and your newest version of the script 0.2.1

Here's the exact input and output for when I try to extract the B.Font files I attached in the first post; are you saying you've been able to extract them no problem?

Code: Select all

E:\QuickBMS>quickbms_4gb_files -o u.bms "B.Font.toc"

QuickBMS generic files extractor and reimporter 0.8.0 (64bit test)
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Mar 12 2017 - 14:58:46)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- open input file E:\QuickBMS\B.Font.toc
- open script u.bms
- set output folder .

  offset           filesize   filename
--------------------------------------
- enter in folder E:\QuickBMS
  coverage file 0     0%   0          36872      . offset 0000000000000000
- open input file E:\QuickBMS\B.Font.toc
- enter in folder E:\QuickBMS
- open input file E:\B.Font.cache
  0000000000000000 993        /EE/Types/ScriptCommands/SampleScripts.lua
  0000000000000000 7824       /EE/Types/ScriptCommands/GraphVizScript.lua
  0000000000000000 1417       /Tests/Scripts/ArgHidingTest.lua
  0000000000000000 1166       /Tests/Scripts/HookCrash.lua
  0000000000000000 3205       /Tests/Scripts/ArgsTest.lua
  0000000000000000 998        /Tests/Scripts/Corruption.lua
  0000000000000000 293        /Tests/Scripts/UtilB.lua
  0000000000000000 409        /Tests/Scripts/UtilA.lua
  0000000000000000 438        /Tests/Scripts/UtilC.lua
  0000000000000000 9942       /Tests/Scripts/Tests.lua
  0000000000000000 794        /Tests/Scripts/ReentrantHandler.lua
  0000000000003043 195        /Tests/Scripts/ScopeTestA.lua
  0000000000003106 195        /Tests/Scripts/ScopeTestB.lua
  0000000000000000 1590       /Tests/Scripts/ScopedDataDriven.lua
  0000000000000000 2688       /Tests/Scripts/WeakPtrDerivation.lua
  0000000000000000 723        /Tests/Scripts/LightTest.lua
  0000000000000000 1933       /EE/Menus/Progress.lua
  0000000000000000 1302       /Keystone/Scripts/Challenges/Assist.lua
  0000000000000000 1161       /Keystone/Scripts/Challenges/AssistSupport.lua
  0000000000000000 2310       /Keystone/Scripts/Challenges/MultiKill.lua
  0000000000000000 2326       /Keystone/Scripts/Challenges/KillStreak.lua
  0000000000000000 1722       /Keystone/Scripts/Challenges/KillStreakMelee.lua
  0000000000000000 826        /Keystone/Scripts/Challenges/KillWhoKilledYou.lua
  0000000000000000 440        /Keystone/Scripts/Challenges/KillFromTheGrave.lua
  0000000000000000 593        /Keystone/Scripts/Challenges/Suicide.lua
  0000000000000000 1670       /Keystone/Scripts/Challenges/CapturePointCombatAttacker.lua
  0000000000000000 2411       /Keystone/Scripts/Abilities/ActivateUltimate.lua
  0000000000000000 16751      /Keystone/Scripts/ChampionVisibilityBarkScript.lua

Error: incomplete input file -1:
       Can't read 367 bytes from offset 0000000000004000.
       Anyway don't worry, it's possible that the BMS script has been written
       to exit in this way if it's reached the end of the archive so check it
       or contact its author or verify that all the files have been extracted.
       Please check the following coverage information to know if it's ok.

  coverage file -1  100%   16384      16384      . offset 0000000000004000

Last script line before the error or that produced the error:
  88  log FNAME 0 SIZE MEMORY_FILE

[aluigi]

Ah ok, it gave no problem with the last sample you provided.

Script 0.2.2, now it works perfectly (it checks even the size of the decompressed file so sure at 100%)

[sephirothxi]

Works perfectly — thank you so much! This is a huge help