REIMPORT2 mode - no size limits

[aluigi]

Just a quick announcement about an alternative reimport mode implemented in QuickBMS 0.8.2a.

Now in the quickbms package you will find a new reimport2.bat file that activates this new mode.
REIMPORT2 is no longer limited by the size and compressed size of the reimported files, if the file is bigger it will be appended at the end of the archive.

The usage is the same of reimport.bat, the only difference for the user is that you will no longer see the usual "infamous" error:

Code: Select all

Error: file "???"
       the reimport option acts as a reimporter and so you cannot reinsert a
       file if it's bigger than the original otherwise it will overwrite the
       rest of the archive or cannot be loaded correctly:

         new size: ? uncompressed
         old size: ? uncompressed

- do you want to skip this file? (y/N/force)

It doesn't mean that your edited archive will work "for sure" with your game, it just offers a way to bypass 2 important limits of the standard reimport mode:

• size of the files
• TOC information with some archives/games, in fact the offset, size and compressed size fields will be rewritten

Technical explanation:

Example of input archive

Code: Select all

###112233################DATADATADATA#######
   | | |                 |
   | | |                 file
   | | size
   | compressed size
   offset

Standard reimport mode

Code: Select all

###112233################DATADATADATA#######
                         |          |
                         | new file |

New reimport2 mode

Code: Select all

###112233################DATADATADATA#######DATADATADATADATADATA
   | | |                                    |                  |
   | | |                                    | new bigger file! |
   | | new size
   | new compressed size
   new offset

As usual you will find all the info and limits (don't expect miracles) in section 3 of http://aluigi.org/papers/quickbms.txt

Feel free to post your positive and negative feedback about reimport2 here in this topic

[sigroon365]

Great works! Amazing!

[Drawing]

thanks a lot Sensei

[Delutto]

Amazing news!!! With this, QuickBMS will become the definitive game moding/hacking tool.
Great work master!

[hackspeedok]

Amazing !!! Great work. Thanks Aluigi

[aluigi]

Not sure if it may be useful, but for the next version of quickbms I have in mind a sort of "install"/"uninstall" option that uses the REIMPORT2 mode.
Let's say you make a translation project that modifies 10 files of the archive and you want to distribute it easily and allow the user to apply it as a "patch" that he can remove if necessary.
Technically that would be easily possible by saving the original offset/size/zsize fields and appending all the new files (even those smaller than the originals) keeping the original files untouched, so it's easy to revert the modifications by restoring the original values and "cutting" the archive to its original size.

Anyway, a part from this idea, currently I'm mainly interested in what games work and don't work with reimport2, just for curiosity and "statistics".
Waiting your WORK / DOESN'T WORK list

*edit*
Right, the archives with a TOC located at the end of the archive (the "tail") can't work in reimport2 with files bigger than the originals since the TOC is no longer located at the end of the archive.
Just for statistics, In my collection of over 1800 scripts about 70 have the TOC at the end of the archive.

[Anexenaumoon]

Awesome! So there's no longer a need for that "5GB quickbms" executable?

[aluigi]

quickbms_4gb_files.exe is necessary for archives bigger than 4 Gb because it uses 64bit variables instead of 32bit.
It's also necessary in specific scripts that require 64bit operations.

[Delutto]

Right, the archives with a TOC located at the end of the archive (the "tail") can't work in reimport2 with files bigger than the originals since the TOC is no longer located at the end of the archive.

Well, how about a parameter to set a offset at the end of file to reinsert the content? So, this way we can insert the mod file before the TOC or after the last content of the file.

Anyway, a part from this idea, currently I'm mainly interested in what games work and don't work with reimport2, just for curiosity and "statistics".
Waiting your WORK / DOESN'T WORK list

Unreal Engine 4 PAK files have a TOC in the end of file, so should not work.

[aluigi]

Well, how about a parameter to set a offset at the end of file to reinsert the content? So, this way we can insert the mod file before the TOC or after the last content of the file.

The only work-around I can see about the TOC-tails is simply doing reimport2 as-is now and then copying the whole TOC at the end of the file, but it will never work with the archives that write the offset of the TOC at the end (many):

goto -8
idstring "SIGN"
get TOC_OFFSET long
goto TOC_OFFSET

That would work only with archives that don't have a TOC offset, like those in which the starting of the TOC is calculated from the number of files or from the size of the TOC, for example: savepos OFFSET ; xmath OFFSET "OFFSET - (FILES * 8)" ; goto OFFSET.
And even in that lucky rare case there is still the problem that quickbms can't magically guess where the TOC starts and ends.

Long story short: no solution or work-around.

No problems for those archives with TOC at the end and TOC_OFFSET at the beginning since doesn't matter where the TOC and the files are located, that already works.

Ah in the "about 70" scripts I guessed as TOC-tail in my previous post I forgot to calculate those having the TOC offset at the beginning and those using "get OFFSET asize ; math OFFSET - ?" instead of "goto -?", so the number is a bit higher, probably still under 100 which is basically "nothing" compared to 1900 scripts.

[shadow_lonely]

Nice! That's great!

[aluigi]

In quickbms 0.8.3 I have started to add a feature that tells the user if the archive is invalid for reimport2, it will be improved over time because currently it simply checks if the scripts uses "goto -NUM" (TOC at the end).

I'm not sure if it's a good idea to zero the bytes where the original file is located in the archive, in theory yes because that space is no longer used and the archive will compress better if you want to distribute it.
I preferred to wait, let me know if you find something wrong in doing that.

I don't think I will implement the patch/unpatch option I mentioned some posts above because currently I have not enough statistics to know if it would be worth or not.

[sigroon365]

Test result - Doesn't work on Hitman 2016.

[aluigi]

I guess the problem with hitman is caused by:

• ZSIZE which is a result of various "&"
• SIZE coming from a memory file
• HASH may be checked by the game at runtime, but I don't know if it's the hash of name (probably) or file (I doubt)

[Sajjad_Rahim]

i just tried REIMPORT2 with the evil within 2 but game still crashing... i think because game use ptr for offset and size and pkr for compressed file. any solution?

[sigroon365]

I guess the problem with hitman is caused by:

• ZSIZE which is a result of various "&"
• SIZE coming from a memory file
• HASH may be checked by the game at runtime, but I don't know if it's the hash of name (probably) or file (I doubt)

Thanks for your reply.

[Graveyard]

Excelent!!!

[Delutto]

Waiting your WORK / DOESN'T WORK list

REIMPORT 2 doesn't work with yoyogames.bms, more specifically with STRG(Text) file. (Or am I doing something wrong)
I think Slog is the problem(as descripted in section 3 of quickbms.txt), I tested some binary STRG dump with a following script:

Code: Select all

get STRINGS long
for i = 0 < STRINGS
   get DUMMY long
next i
for i = 0 < STRINGS
   get STRING_SIZE long
   math STRING_SIZE + 1
   slog "Textos.txt" -1 STRING_SIZE
next i

STRG.7z

[aluigi]

Offset -1 means that there is no absolute offset, so the strings are stored sequentially and therefore it's not possible to store a string longer than the original

[GHOST DEAD]

in Force mod, archive will work with a new size or not? (for those script can be repack(inject) a file in archive)