Shadow of Mordor archive editing and files appending via reimport2

[Mana]

Is there any way to force appending something in reimport 2 mode? I'm pretty sure I've figured out how to add new files to the end of the Shadow of War .arch06 archives, but I'd need to have quickbms append them so that they're compressed. I'd then manually edit the file to add the appropriate file name and file size values (just before each file in the archive theres a value for the size of that file). Reimport2 doesn't work with these files because they're sequential, and I can't force it with that mode.

[Mana]

While I wait to hear back, I'm going to test my theory by attempting to add a new file to the end of an archive. I'll edit this comment with the results.

EDIT: Wasn't as easy as I'd hoped it would be, but initial results show promise. Haven't managed to append the file yet, as the format was more complicated than it first appeared. I am certain that appending a new file is possible with the right knowledge. Aluigi could almost certainly do it, I just don't know the format well enough yet. I still need to be able to force reimport2.bat to append a compressed file to the end of it.

[aluigi]

No, you can't add new files

[Mana]

Why not? Whats special about this format compared to most other archives that can have files added? I don't mean automatically using just quickbms and nothing else, but manually through hex editing. I just need to force the reimport to get the binary data I need, which I would then manually add into another copy of the archive with the appropriate adjustments to the header and file offsets. I've tried minor adjustments to the archives before, and provided the offsets are correctly altered the game loads them fine.

[aluigi]

I don't mean automatically using just quickbms and nothing else

Why are you posting a game-specific question in a topic dedicated to a specific feature of quickbms?

You are off-topic, the topic of that game is viewtopic.php?f=9&t=5080

[Mana]

Because it will require forcing the quickbms reimport2 mode to do it. At least for now. My question was is there any way to force reimport2 to attempt it. If not could I put in a feature request for that?

[aluigi]

Once you correctly modify the TOC (table containing information for each file) adding the reference to the new file using offset and sizes equal than 0, quickbms will do everything automatically.

[Mana]

Oh I see. Could you clarify what you mean by sizes equal to 0? Did you mean greater than 0? Thanks for the answer, I'll move to the games specific thread now. In order to get the correct sizes for the files (for example, a modified mesh or altered file), I'd need to reimport it into something wouldn't I? I feel like I'm misunderstanding something.

[aluigi]

When you run quickbms with the -V option it displays all the fields read from the format, it's a sort of debugging option which is extremely useful while writing scripts.

Depending by the script, a size equal than 0 may force the script to skip the file, so let's say (as universal rule) to use 1 for all the 2+1 fields involved:
OFFSET
SIZE
ZSIZE (the compressed size is sometimes optional, depends by the format)

So if all these fields for one file are equal than 1, then quickbms in reimport2 will be automatically forced to append the file at the end of the archive and rewrite the new 3 fields over the original.

Please note that I'm expecting you perfectly know things like TOC size (you can't add new files if you can't increase the TOC), endianess, size of the fields and everything else necessary to do what you are trying to do (and there are big chances that's not possible at all due to the TOC size).
If you have no experience with this stuff please stop, or feel free to continue but don't ask me support (eh limited time).

Now I would like to move these OT posts in a new topic in another section if that's ok for you.

[Mana]

Of course. Thanks for being so accommodating. I've got experience with this stuff, but my knowledge is a bit iffy. I'm an artist initially, programming stuff takes me a bit to learn. I know most of the things you listed, though not necessarily how they work in this particular format just yet. My eventual plan is to add "new" files to these archives (or new archives completely) to allow for model swapping. Should be really cool if I can get it to work.

[aluigi]

Adding new files with the limitations of the TOC size is something painful.
But, if the game accepts new archives (for example N archives with different names instead of one big monolytic archive, it was possible with Telltale games), you may opt for something like copying one existent archive and editing its entries replacing the existent ones with those of the new files you want to add.
But even in that case it's a quite big work to do manually through hex editing.

[Mana]

Yeah, the archives the game loads are read from a plain text file. My current plan is to manually add in just the files I need to replace into a new archive and load that. Requires a bunch of manual work, but its looking promising. I wasn't actually expecting it to work out but so far its going well.

EDIT: Incidentally, I have a very odd "bug" resulting from doing this. The T in the LTAR of the first four bytes gets replaced with 00 for some reason. I just manually put the T back, but I can't imagine why that happens.

[aluigi]

EDIT: Incidentally, I have a very odd "bug" resulting from doing this. The T in the LTAR of the first four bytes gets replaced with 00 for some reason. I just manually put the T back, but I can't imagine why that happens.

Right, I forgot about that.
Basically reimport2 automatically zeroes the old location of the file for debugging while testing the edited archive and allowing the compression of the whole archive if the modder wants to distribute it.
If ZSIZE/SIZE was zero there was no problem but since you are forced to use a positive number (due to the script requiring it) then you will have this side-effect, just remember to manually restore the 'T'

[aluigi]

I have splitted the posts in a new topic so you can ask and provide information and updates without any OT problem

[Mana]

Thanks. This whole thing has been going so much better than I'd hoped. The folder structure is going to be one of the hardest bits I think. The game will be searching for these models in a certain folder structure. I'll need to create that in my new archive. I've looked at some existing archives to try and wrap my head around how the folders get extracted. Are they treated like files or something?

[aluigi]

If your archive is an LTAR one, then this is the part of the script you have to consider:

Code: Select all

        get VER long
        get NAMES_SIZE long
        get MAIN_FLAGS long
        get FILES long
        getdstring DUMMY 0x1c

        savepos OFFSET
        log MEMORY_FILE OFFSET NAMES_SIZE
        math OFFSET + NAMES_SIZE
        goto OFFSET

        for i = 0 < FILES
            get NAME_OFF long
            get OFFSET longlong
            get ZSIZE longlong
            get SIZE longlong
            if VER >= 4
                get DUMMY byte  # 1
                get FLAGS byte  # usually 9 for compressed chunks
            else
                get FLAGS long  # usually 9 for compressed chunks
            endif

            goto NAME_OFF MEMORY_FILE
            get NAME string MEMORY_FILE
            string NAME R= "_" "/"

            if ZSIZE > 0
                if SIZE != ZSIZE    # or FLAGS & 1 (FLAGS is usually 9)
                    ...
                else
                    log NAME OFFSET SIZE
                endif
            endif
        next i

I don't remember if folder are archived anyway, maybe with ZSIZE zero, or not.
The script says that the path delimiter is the underscore and not the slash/backslash, so if you have a file with full name FOLDER1\FOLDER2\FILE.TXT you have to set it as FOLDER1_FOLDER2_FILE.TXT
Remember to set FLAGS to zero or 8 (you must try both) just in case the game enables compression when it's set otherwise.
The best suggestion I can give you is to make tons of tests with different settings.
The archive is probably not very complex and, apparently, there is only one field which is not parsed (DUMMY 0x1c) so you may opt for recreating it from scratch instead of editing existing ones.

[Mana]

Thanks. Doing lots of tests is my preferred method of learning these things. I'll try using the names to create the folder structure. The other option is that instead of mimicking the folders I instead alter the game to look for the files in a different place. If I can't make any headway with the folders I'll try that out.

[Mana]

Seems to have worked, I don't understand how the tool handles files that have an underscore in their name, but I don't need to worry about that for now. I'll keep testing things out, see if mimicking the file structure is enough to get the game to load it in place of the original or not.

EDIT: I've tried a bunch of different files, but haven't been able to replace anything yet. The game isn't crashing, so I'm pretty sure it just means I'm either naming the file wrong, putting it in the wrong file structure, or something similar to that. Tracking down the exact file I want to replace is proving very difficult. I've noticed that some files have underscores in their names, which will be problematic. I'll continue working on this tomorrow. Hopefully I can figure out what I'm doing wrong.

[Mana]

I have a question, not sure if I should ask here or in the general SoW thread. The .BNDL files seem like they're supposed to be openable with the quickbms script, but every one I've tried has failed. Are they supposed to be opened?

[aluigi]

Yes, the BNDL format is supported.
What error you get?
Please upload a sample or a the files generated by http://aluigi.org/bms/filecutter.bms