XMemDecompress code

[puggsoy]

I recently discovered that QuickBMS supports XMemDecompress, the compression method used in XNB files for XNA games. I was curious to look at the code itself but couldn't find a find that seemed to be related to it in the source.

Would you be able to direct me towards where it is in the source?

[Ekey]

source\libs\mspack\

[puggsoy]

Thanks Ekey I've heard that the method is also known as LZX so I assume that the relevant code is in lzx.h and lzxd.c.

[aluigi]

As far as I know the code in mspack is not 100% compatible with xmemdecompress, I remember to have tried it some months/years ago.

[Ekey]

Well, if you want i can share module with defines the Xbox Compression APIs

[aluigi]

The Xbox SDK is the only place where is located the xmemdecompress code.
I'm not aware of open source versions, I heard that someone probably tried to do something for a tool related to a game (Unreal engine?) but I don't remember and if I tried it, it didn't work.
So check xcompress.h and xcompress.lib if you want to use xmemdecompress.

[puggsoy]

How does QuickBMS use XMemCompress? The XNB script works perfectly on some files I tried.

[aluigi]

xcompress.lib

[beedy]

I ask my question in this topic.

I need help to re-compressing Xbox 360 NHL Legacy chunklzx compressed texture packs using quickbms. I copied attached part of fightnight.bms to get compression algorithm and I can now decompress files correctly but when I re-importing modified textures with quickbms it appears problems with a changed sizes.

Decompressed texlibs are same sized and every textures inside texlibs are also same sized, but compressed file sizes should be different sizes.
I have no idea what the lzx-compression do for files during the compression process, but the result of re-importing with quickbms is not perfect.

Is there way to compress the textures to chunklzx with xmemdecompress algorithm to a new file using quickbms or other way instead of re-injecting it inside original file?

Here you can see the result of re-importing/injecting using quickbms
http://imgur.com/a/kJnhJ

Attachment contains the sample file.

Code: Select all

# Fight Night Champion / SSX Xbox 360 (*.big) (script 0.3.4)
# script for QuickBMS http://quickbms.aluigi.org

get NAME basename
get EXT extension
string NAME += "_unpacked."
string NAME += EXT

endian big
    comtype xmemdecompress
    savepos TMP
    goto OFFSET
    idstring "chunklzx"
    get DUMMY long
    get FULLSIZE long
    get SIZE long
    get CHUNKS long
    get DUMMY long
    get DUMMY long
    get DUMMY long
    get DUMMY long
    log MEMORY_FILE3 0 0
    append
    savepos OFFSET
    for j = 0 < CHUNKS
        math OFFSET x= 0x8
        for # very lame, made on the fly
            math T = OFFSET
            math T %= 0x10
            if T == 8
                break
            endif
            math OFFSET += 8
        next
        goto OFFSET

        get ZSIZE long
        get ZIP long
        savepos OFFSET
        if ZIP == 3
            clog MEMORY_FILE3 OFFSET ZSIZE SIZE
        elif ZIP == 4
            log MEMORY_FILE3 OFFSET ZSIZE
        else
            print "unknown ZIP type %ZIP%, contact me!"
            cleanexit
        endif
        math OFFSET += ZSIZE
    next j
    append
    get MYSIZE asize MEMORY_FILE3
    if MYSIZE != FULLSIZE
        print "Alert: MEMORY_FILE3 %MYSIZE% != %FULLSIZE%"
    endif
    log NAME 0 FULLSIZE MEMORY_FILE3

[aluigi]

If you just want to compress one file:

Code: Select all

comtype xmemlzx_compress
get SIZE asize
clog "new.dat" 0 SIZE SIZE

[beedy]

Thank you aluigi for your answer. Yes I want to compress one file and then inject the file onto .big archive and then injecting the .big archive to original game .iso file using Xbox image browser. a File compression is the only problem to get perfect texturemods in game. More about that

When I compress uncompressed file with this:

Code: Select all

comtype xmemlzx_compress
get SIZE asize
clog "new.dat" 0 SIZE SIZE

I get:
http://imgur.com/a/Hszrt
Should be:
http://imgur.com/a/zxF5a

How to fit that with chunklzx script? It should need parse a header for compressed file. 00 3B 10 00 is uncompressed file size, 00 00 00 0F is a number of chunks, what is 40 00 00 (get SIZE long)? I will keep learning but I'm very thankfully if you can help me whit that.

[aluigi]

Basically what you want is a rebuilder because chunklzx is not a compression, it's a file format with its fields and rules.
Additionally it's even not very "standard" in the sense that some fields are unknown and I used some work-arounds.
I don't write rebuilders so I can't help.

The script doesn't support reimporting for various reasons:
- the file is split in chunks, in theory quickbms can do the job but in practice users report ever problems
- usage of MEMORY_FILE to reassemble the file (but this is something that can be fixed)
- compression algorithms that will crash the game like lz4 viewtopic.php?p=5500#p5500 (I don't remember if xmem may give the same problems)

The only thing which is far from being a solution and still doesn't solve the possible third problem is modifying the script to extract the single chunks, reimporting will work well in that way. Let me know if you want to try and if that's ok for your needs.

The alternative is writing a rebuilder by yourself.

[beedy]

Thank you, I have never written any programs so it's quite challenging at this situaition. However I can understand hex code a little so it's a good start. I copied first 0x40000 (got this from chunklzx offset 0x10) bytes from uncompressed .rx2 to a new file. Then I do that xmemlzx_compress and first bytes of new.dat are identical with original chunklzx file. \o/

original chunklzx file starts at 0x30
http://imgur.com/3Gbsiwk
New compressed .dat file:
http://imgur.com/a/jkqyW

[beedy]

Hi Aluigi!
I got 15 chunks of compressed data to a "new.dat" file using this:

Code: Select all

endian big
CHUNKS = 15
OFFSET = 0
append
for i = 0 < CHUNKS
   if OFFSET > 0x340000
      SIZE = 0x31000
   else
      SIZE = 0x40000
   endif
   comtype xmemlzx_compress
   clog "new.dat" OFFSET SIZE SIZE
   math OFFSET += 0x40000
next i

I got chunks identical with original chunklzx file.

Original chunklzx has two long fields before every chunks: compressed size of chunk for example 00 01 00 2E and unknown 00 00 00 03. Could you help me how to add these fields before every chunks to "new.dat"?

[aluigi]

Code: Select all

comtype xmemlzx_compress
endian big
math CHUNK_SIZE = 0x40000
get SIZE asize

set NAME string "new.dat"
log NAME 0 0
for OFFSET = 0 != SIZE
    math TMP = SIZE
    math TMP - OFFSET
    if TMP < CHUNK_SIZE
        math CHUNK_SIZE = TMP
    endif

    clog MEMORY_FILE OFFSET CHUNK_SIZE CHUNK_SIZE
    get CHUNK_ZSIZE asize MEMORY_FILE
    putvarchr MEMORY_FILE2 0 CHUNK_ZSIZE long
    putvarchr MEMORY_FILE2 4 3           long
    append
    log NAME 0 8           MEMORY_FILE2
    log NAME 0 CHUNK_ZSIZE MEMORY_FILE
    append
next OFFSET + CHUNK_SIZE

[beedy]

Thank you!