KINGDOM HEARTS HD (Windows) - *.hed+*.pkg

[tbmq008]

Code: Select all

# KINGDOM HEARTS HD 1.5 + 2.5 ReMIX (Microsoft Windows)
# KINGDOM HEARTS HD 2.8 Final Chapter Prologue (Microsoft Windows)

math is_hed = 0
math is_pkg = 0

get ext extension
if ext == "hed"
   math is_pkg = 1
   callfunction handle_hed_pkg 1
elif ext == "pkg"
   math is_hed = 1
   callfunction handle_hed_pkg 1
endif

startfunction handle_hed_pkg
   open FDDE "hed" is_hed
   open FDDE "pkg" is_pkg
   get hed_size asize is_hed
   xmath hed_files "hed_size / 0x20"
   for i = 0 < hed_files
      set name string ""
      getdstring hash 0x10 is_hed
      string hash_in_str b hash
      putvarchr hash_in_str 32 0
      get file_offset longlong is_hed
      get file_size1 long is_hed
      get file_size2 long is_hed
      set name string hash_in_str
      goto file_offset is_pkg
      get file_size3 long is_pkg
      get sub_entries long is_pkg
      get compressed_file_size1 long is_pkg
      get file_info3 long is_pkg
      if sub_entries == 0
         string name + "."
         math file_offset + 16
         if compressed_file_size1 <= 0xfffffff0
            log name file_offset compressed_file_size1 is_pkg
         else
            log name file_offset file_size3 is_pkg
         endif
      else
         string name1 p "%s/init." name
         xmath helper "16 + (sub_entries * 0x30)"
         math file_offset + helper
         if compressed_file_size1 <= 0xfffffff0
            log name1 file_offset compressed_file_size1 is_pkg
            math file_offset + compressed_file_size1
         else
            log name1 file_offset file_size3 is_pkg
            math file_offset + file_size3
         endif
         for j = 0 < sub_entries
            string name2 p "%s/%u." name j
            getdstring sbe_name 0x20 is_pkg
            get sbe1 long is_pkg
            get sbe2 long is_pkg
            get sbe3 long is_pkg
            get sbe4 long is_pkg
            if sbe4 <= 0xfffffff0
               log name2 file_offset sbe4 is_pkg
               math file_offset + sbe4
            else
               log name2 file_offset sbe3 is_pkg
               xmath sbe3_1 "sbe3 && 0x10"
               math file_offset + sbe3_1
            endif
         next j
      endif
   next i
endfunction

this script can only extract files as-is, as a consequence it cannot handle any decompression stuff whatsoever. this is why you get some garbled files.

[admassacration]

Hello tbmq008.
I have a project to translate the game into Portuguese-BR and I am looking for tips on how to extract the texts from the game.
Use the bms with the script above with the files "SettingMenu.hed" and "SettingMenu.pkg".
When extracting, I have the files ".sed", ".l2d" and ".dat".
Any ideas on how to extract texts from menus and dialogues?

Thanks.

[tbmq008]

".sed" files are simply sound data.
as for the rest i have no idea, i went in and wrote the script in "blind mode".

[GrzybDev]

As far as i can see all locale files starts with "@CTD", then some metadata and then UTF-16 encoded strings.
Replacing strings should be easy if someone figures out that metadata before strings

I've uploaded file where you can see results of "translating" this file to blah language (it's the settings menu in launcher),

[GrzybDev]

Good news, i managed to somewhat reverse engineer that format and now i can successfully extract strings (at least from file i sent previously)

Here's Python code that i used to extract file to ".po" file for translation

Code: Select all

import struct, polib

locFile = open("test.dat", "rb")

if struct.unpack('cccc', locFile.read(4)) == (b'@', b'C', b'T', b'D'):
    _, = struct.unpack('h', locFile.read(2))  # Version?
    _, = struct.unpack('i', locFile.read(4))  # Padding

    unknown, = struct.unpack("i", locFile.read(4))
    locEntries, = struct.unpack("h", locFile.read(2))
    addrBlockOffset, _, = struct.unpack("hh", locFile.read(4))
    hashBlockOffset, _, = struct.unpack("hh", locFile.read(4))
    strBlockOffset, _, = struct.unpack("hh", locFile.read(4))
    _, = struct.unpack("i", locFile.read(4))  # Padding

    print("Entries: " + str(locEntries))
    print("----------------------")
    print("Address block offset: " + str(hex(addrBlockOffset)))
    print("Hash Block Offset: " + str(hex(hashBlockOffset)))
    print("String block offset: " + str(hex(strBlockOffset)))
    print("----------------------")
    
    locFile.seek(addrBlockOffset)

    offsets = []

    for i in range(locEntries):
        locID, setID, = struct.unpack('HH', locFile.read(4))
        offset, unknown, = struct.unpack("hh", locFile.read(4))
        offsets.append(offset)

    locFile.seek(strBlockOffset)

    i = 0
    poFile = polib.POFile()

    for offset in offsets:
        locFile.seek(offset)
        lastChar = b""

        while lastChar != b"\0":
            lastChar = locFile.read(1)
            locFile.read(1)

        strLen = locFile.tell() - offset - 2
        text = u""

        if strLen != 0:
            locFile.seek(offset)

            rawStr = locFile.read(strLen)
            text += rawStr.decode('utf-16')

        print(str(i) + " - " + text)

        poFile.append(polib.POEntry(
            msgid=text,
            msgstr="",
            occurrences=[("test", str(i))]
        ))

        i += 1
    
    poFile.save("test.po")

Re-importing shouldn't be a problem if output file will be no-longer than original file, but as far as i can see archive itself is pretty "flexible"

Here's proof, that this scripts outputs valid .po files:

Zrzut ekranu 2021-04-05 142442.png

[GrzybDev]

Okay, i finished both exporter and reimporter for locale files. It seems to work (i can freely modify texts inside settings menu), can someone help me identifying font files? And helping me replacing/adding characters?


Exporter:

Code: Select all

import struct, polib

locFile = open("test.dat", "rb")

if struct.unpack('cccc', locFile.read(4)) == (b'@', b'C', b'T', b'D'):
    _, = struct.unpack('h', locFile.read(2))  # Version?
    _, = struct.unpack('i', locFile.read(4))  # Padding

    unknown, = struct.unpack("i", locFile.read(4))
    locEntries, = struct.unpack("h", locFile.read(2))
    addrBlockOffset, _, = struct.unpack("hh", locFile.read(4))
    hashBlockOffset, _, = struct.unpack("hh", locFile.read(4))
    strBlockOffset, _, = struct.unpack("hh", locFile.read(4))
    _, = struct.unpack("i", locFile.read(4))  # Padding

    print("Entries: " + str(locEntries))
    print("----------------------")
    print("Address block offset: " + str(hex(addrBlockOffset)))
    print("Hash Block Offset: " + str(hex(hashBlockOffset)))
    print("String block offset: " + str(hex(strBlockOffset)))
    print("----------------------")
    
    locFile.seek(addrBlockOffset)

    offsets = []

    for i in range(locEntries):
        locID, setID, = struct.unpack('HH', locFile.read(4))
        offset, unknown, = struct.unpack("hh", locFile.read(4))
        offsets.append(offset)

    locFile.seek(strBlockOffset)

    i = 0
    poFile = polib.POFile()

    for offset in offsets:
        locFile.seek(offset)
        lastChar = b""

        while lastChar != b"\0":
            lastChar = locFile.read(1)
            locFile.read(1)

        strLen = locFile.tell() - offset - 2
        text = u""

        if strLen != 0:
            locFile.seek(offset)

            rawStr = locFile.read(strLen)
            text += rawStr.decode('utf-16')

        poFile.append(polib.POEntry(
            msgid=text,
            msgstr=""
        ))

        i += 1

    poFile.save("test.po")

Reimporter:

Code: Select all

import polib, struct

origFile = open("test.dat", "rb")
newFile = open("BEB2DD13552A07B1BAC2E1FC4DCD6837.dat", "wb")

origFile.seek(0x18)  # strBlockOffset location
strBlockOffset, = struct.unpack("h", origFile.read(2))
origFile.seek(0)

translated = polib.pofile("test.po")
newFile.write(origFile.read(0x20))

offset = strBlockOffset

for string in translated:
    strLen = 0

    if string.msgstr == "" and string.msgid == "":
        strLen = len("\0".encode("utf-16-le"))
    elif string.msgstr == "":
        strLen = len(string.msgid.encode("utf-16-le")) + 2
    else:
        strLen = len(string.msgstr.encode("utf-16-le")) + 2

    newFile.write(origFile.read(4))
    newFile.write(struct.pack("h", offset))
    origFile.read(2)
    newFile.write(origFile.read(2))

    offset = offset + strLen

newFile.write(origFile.read(strBlockOffset - origFile.tell()))

for string in translated:
    if string.msgstr == "" and string.msgid == "":
        newFile.write("\0".encode("utf-16-le"))
    elif string.msgstr == "":
        newFile.write(string.msgid.encode("utf-16-le"))
        newFile.write("\0".encode("utf-16-le"))
    else:
        newFile.write(string.msgstr.encode("utf-16-le"))
        newFile.write("\0".encode("utf-16-le"))

origFile.seek(-4, 2)
newFile.write(origFile.read(4))
origFile.close()
newFile.close()

[admassacration]

GrzybDev, good afternoon.
Is this Python script used in quickbms to export and import files?
Thank you.

[Dark_Ansem]

Nice does this allow full unpack of the directories and access to the resources?

[GrzybDev]

Code that i posted here was only proof of concept

I'm currently writing an tool, for unpacking and packing all archive files (and decompiling and compiling of locale files, i temporarily removed locale tool because "encrypted" archives contains format which is a little bit diffirent from not-encrypted ones, but it will be soon available)

Code: https://github.com/GrzybDev/KingdomHeartsTool

Currently this tool are able to unpack all encrypted and not encrypted archives /w compression etc., pack tool might work only on files which are originally not-encrypted and not-compressed (because i'm not saving some important data, and i'm not able to create byte-perfect archives... yet)

Anyway, if anyone really want to use that tool asap, even if pack feature is not complete here is example on how to use my tool:
Unpack archive:

Code: Select all

python -m khtool -v -u "B:\Kingdom Hearts HD 1 5 and 2 5 ReMIX\Image\en\bbs_first.hed" -o output

Repack archive:

Code: Select all

python -m khtool -v -p output/bbs_first.json -o output/bbs_first

Edit: Now i finished both unpack and pack tool, so you can freely repack all game archives (if you use both unpack and pack tool on the same set of data, pack tool will create byte-perfect archive)

[guitutilo]

I am trying to translate the menu, the menu file is Image / SettingMenu.hed I used the following method:
python -m khtool -v -u "C:\Games\Kingdom Hearts HD 1 5 and 2 5 ReMIX\Image\SettingMenu.hed" -o output\

Then:
python -m khtool -v -l "output/SettingMenu/*.ctd" -o traduzir/
I translated the file BEB2DD13552A07B1BAC2E1FC4DCD6837.po because this file is from the menu in English. I translated:

msgid "Quit Game"
to:
msgid "Sair do Jogo"

Then:
python -m khtool -v -l "traduzir/output/SettingMenu/*.po" -o final/

I copied the files from the final folder and replaced the one in the output / SettingMenu folder and executed the command:
python -m khtool -v -p output/SettingMenu.json -o output/SettingMenu

I copied the files SettingMenu.hed and SettingMenu.pkg to the Image folder, but it didn't work

[GrzybDev]

There's small problem with offset calculating, will fix it asap

[guitutilo]

When you fix it, will you notify me here on the forum or will I see it for the last commit there in git?

[GrzybDev]

The issue is now fixed, my bad because of old code

I see that you translated msgid entry...
Dude, you never translated anything, before you start READ THE MANUAL https://www.gnu.org/software/gettext/ma ... Files.html

[guitutilo]

Now it's working. I managed to translate some parts of the menu and it worked, thanks.

[guitutilo]

I did an accentual translation, and it didn't work, how could this be done or not?
The word is: Exibição

[GrzybDev]

There's no problem with my tool, it's a problem because your language uses characters that this game do not understand (hence ? instead of chars)
You will need to replace already existing characters in font file, with characters you want.

[guitutilo]

Could you explain to me how to do this? I don't know how to change to accept Brazilian characters.

It is strange not to accept the word ã, but the word õ accepts

[GrzybDev]

It is not a question for me, i only made a tool which allows to compile and extract strings from game files
You need to do your own research on how to modify font files, because i'm not interested in graphic files (yet)

My team provided me list of characters from font, so it is possible:

[guitutilo]

Ok I understand, I just want to know which file to change the font?

[GrzybDev]

I don't know, as i stated i only care about text files. And it's obviously not one font, but at least 20 (from what i can see)

They can be anywhere, maybe they are in .i2d files, maybe .tm2