Zero escape:999(data.bin)

[zkdldk95]

Hello, I'm trying to translate Zero Escape The Nonary Games but have some problems.

I extracted .sir files from ze1_data.bin, and making any change of data size cause errors in game.

Of course, I used reimport2.bat for reimporting the sir files and adjusted their offsets.

Also, I checked just that exchanging a word is fine. (door -> dddd is printed well)

I don't know what the problem is. Please give some help.
(And what is the 5-byte data at the end of offsets?)

[haibaer]

search old posts. i've explained why reimport2 is not working well.
For some kind of files, you also need to modify the last few bytes which contains a 7-bit encoded (check variable-length quantity on wikipedia) data size.
It has been a while since i finished the localization for my language so i need to check my codes for details if you need.
All questions are welcome

[zkdldk95]

search old posts. i've explained why reimport2 is not working well.
For some kind of files, you also need to modify the last few bytes which contains a 7-bit encoded (check variable-length quantity on wikipedia) data size.
It has been a while since i finished the localization for my language so i need to check my codes for details if you need.
All questions are welcome

Thank you for the answer. I learned the variable-length quantitiy but still have some question.

A file name is 00009f13.sir and [04 08 84 A8 78] is on the last.

I thought [84 A8 78] designates the file size and changed it to binary number.

1000 0100
1010 1000
0111 1000

-> 100 010 1000 111 1000 = 70776(dec) = 0x14478

but the file size is 102KB (105,264 Bytes). I don't know what's going on.

And second, how did you change the fonts to your language?

Thanks again.

[haibaer]

00009f13.sir contains script texts.
0x00 - 0x03: SIR1
0x04 - 0x07: start offset of text index table(0001147C)
0x08 - 0x0B: 0
0x0B - 0x0F: end offset of text index table
0x10 - 0x13: 0
0x14 - 0001147B: texts
0001147C : text index table
...

Each entry of the text index table is a 8-byte absolute offset of the file and each text is a 0-terminated string:
0x00000014: M10a_050_010_02_10
0x00000027: Talk
...
...

If your final script text won't exceed the original size, you can simply modify the index table, adjust the offset of each text item, and fill the rest part of texts with zero.(No need to use reimport2, and I believe aluigi has already provided a bms script to do this automatically)
If not, you need to modify the 7-bit encoded data.

04 -> the first 4-byte of the file
08 -> next 8 bytes
84 A8 70 -> 0x11470 next 0x11470 bytes (to the start offset of text index table)
08 08 08 08 ...... -> Each '08' corresponds to an text index table entry.

00009ec1.sir is a font file, you need to modify these font files if your language contains non-ascii characters. They are stored as 8-bit grayscale bitmaps.

[zkdldk95]

00009f13.sir contains script texts.
0x00 - 0x03: SIR1
0x04 - 0x07: start offset of text index table(0001147C)
0x08 - 0x0B: 0
0x0B - 0x0F: end offset of text index table
0x10 - 0x13: 0
0x14 - 0001147B: texts
0001147C : text index table
...

Each entry of the text index table is a 8-byte absolute offset of the file and each text is a 0-terminated string:
0x00000014: M10a_050_010_02_10
0x00000027: Talk
...
...

If your final script text won't exceed the original size, you can simply modify the index table, adjust the offset of each text item, and fill the rest part of texts with zero.(No need to use reimport2, and I believe aluigi has already provided a bms script to do this automatically)
If not, you need to modify the 7-bit encoded data.

04 -> the first 4-byte of the file
08 -> next 8 bytes
84 A8 70 -> 0x11470 next 0x11470 bytes (to the start offset of text index table)
08 08 08 08 ...... -> Each '08' corresponds to an text index table entry.

00009ec1.sir is a font file, you need to modify these font files if your language contains non-ascii characters. They are stored as 8-bit grayscale bitmaps.

I've tried it and still the error comes out.

Could you check my 00009f13.sir file? (I changed the last text 'door' to 'dooooor' so that the offset table still works.)

Or just give me your localization files, then i'll check it.

[haibaer]

sorry but i did the whole work based on the japanese version so i didn't modify 00009f13.sir.

Here are the possible reasons why your file is not working:
1. AA AA AA at the end of the texts is used to keep the whole address things 4 bytes aligned. you can add one more 'o' to keep the alignment.
2. you've added 5 'o's so 84 a8 70 should now be 84 a8 75, not 84 a8 73.
3. maybe the offset and new file size are not corrected after reimort2.

I also uploaded my 0000119d.sir, a font file that I've modified both the contents and the file size. hope this could help.

I'll test your file on the english version on the weekend if you still have problem

[zkdldk95]

sorry but i did the whole work based on the japanese version so i didn't modify 00009f13.sir.

Here are the possible reasons why your file is not working:
1. AA AA AA at the end of the texts is used to keep the whole address things 4 bytes aligned. you can add one more 'o' to keep the alignment.
2. you've added 5 'o's so 84 a8 70 should now be 84 a8 75, not 84 a8 73.
3. maybe the offset and new file size are not corrected after reimort2.

I also uploaded my 0000119d.sir, a font file that I've modified both the contents and the file size. hope this could help.

I'll test your file on the english version on the weekend if you still have problem

Thank you!! I tried the first solution(adjust number of AA AA AA to keep the alignment) and it works!!

No more error occurs. Have a nice weekend!

[zkdldk95]

Me again.
I reimported your font file and it did't work in japanese mode.

So I have two questions.
1. How did you change the characters? Did you use Python or crystaltiles2?(I'm trying to with crystaltile2 but it's difficult)

2. What are the offsets in the font files? They are pointing middle of a character(so I cant insert more characters)

sorry for many questions. I really want to localize this game. Thank you.

[haibaer]

Me again.
I reimported your font file and it did't work in japanese mode.

So I have two questions.
1. How did you change the characters? Did you use Python or crystaltiles2?(I'm trying to with crystaltile2 but it's difficult)

2. What are the offsets in the font files? They are pointing middle of a character(so I cant insert more characters)

sorry for many questions. I really want to localize this game. Thank you.

I didn't use crystaltiles2.
The index table of the font files uses a relative offset (and divide by 2).
Here's my python code to modify the font file (000011a4.sir). hope this could help.
（The 7-bit encoded part is almost the same as other files If you need modify it.）

Code: Select all

# @brief modify font files so that the game contains our own fonts.
def modify_fonts():
    fin = open('4.sir', 'rb')
    fout = open('000011a4.sir', 'wb')
    rel_poses = []
    old_file = fin.read()
    start = 0
    num = len(CN_ALL_CHAR)
    end = start + num * 4 + 20
    fout.write(b'\x53\x49\x52\x31')
    fout.write(start.to_bytes(8, byteorder='little'))
    fout.write(end.to_bytes(8, byteorder='little'))
    cur_pos = 20

    chs = list(ENCODE_TABLE.keys())
    chs.sort(key = sort_byte)

    for ch in chs:
        rel_poses.append(cur_pos)
        img = to_font(ch, 'fonts', 26)
        fout.write(img)
        cur_pos = cur_pos + len(img)
        if cur_pos % 4 != 0:
            pad = bytearray(cur_pos % 4)
            pad = pad.replace(b'\x00', b'\xaa')
            fout.write(pad)
            cur_pos = cur_pos + cur_pos % 4
    
    # update start and end position
    start = cur_pos
    end = start + num * 4 + 20

    fout.write(num.to_bytes(8, byteorder='little'))
    fout.write((0x0000001b).to_bytes(4, byteorder='little'))
    fout.write((0x00000014).to_bytes(4, byteorder='little'))
    fout.write((0x00000000).to_bytes(4, byteorder='little'))
    for rel in rel_poses:
        fout.write(int((rel - 0x14) / 2).to_bytes(4, byteorder='little'))           # <- the offset
    tab_pad = (16 - (start + 20 + 4 * num) % 16) % 16
    pad = bytearray(tab_pad)
    pad = pad.replace(b'\x00', b'\xaa')
    fout.write(pad)

    fout.write(old_file[-16:])
    fin.close()
    fout.close()
    

[zkdldk95]

Code: Select all

# @brief modify font files so that the game contains our own fonts.
def modify_fonts():
    fin = open('4.sir', 'rb')
    fout = open('000011a4.sir', 'wb')
    rel_poses = []
    old_file = fin.read()
    start = 0
    num = len(CN_ALL_CHAR)
    end = start + num * 4 + 20
    fout.write(b'\x53\x49\x52\x31')
    fout.write(start.to_bytes(8, byteorder='little'))
    fout.write(end.to_bytes(8, byteorder='little'))
    cur_pos = 20

    chs = list(ENCODE_TABLE.keys())
    chs.sort(key = sort_byte)

    for ch in chs:
        rel_poses.append(cur_pos)
        img = to_font(ch, 'fonts', 26)
        fout.write(img)
        cur_pos = cur_pos + len(img)
        if cur_pos % 4 != 0:
            pad = bytearray(cur_pos % 4)
            pad = pad.replace(b'\x00', b'\xaa')
            fout.write(pad)
            cur_pos = cur_pos + cur_pos % 4
    
    # update start and end position
    start = cur_pos
    end = start + num * 4 + 20

    fout.write(num.to_bytes(8, byteorder='little'))
    fout.write((0x0000001b).to_bytes(4, byteorder='little'))
    fout.write((0x00000014).to_bytes(4, byteorder='little'))
    fout.write((0x00000000).to_bytes(4, byteorder='little'))
    for rel in rel_poses:
        fout.write(int((rel - 0x14) / 2).to_bytes(4, byteorder='little'))           # <- the offset
    tab_pad = (16 - (start + 20 + 4 * num) % 16) % 16
    pad = bytearray(tab_pad)
    pad = pad.replace(b'\x00', b'\xaa')
    fout.write(pad)

    fout.write(old_file[-16:])
    fin.close()
    fout.close()
    

It is difficult to understand a little bit but it's ok.

I wonder how can I change a font to byte arrays. ("to_font" in your code might has that function)

I hope this is the last question...

Thanks a lot.

[haibaer]

The font file uses 8-bit grayscale bitmaps.
You can generate grayscale bitmaps from ttf/otf fonts using any font engine. For me I use freetype. (You can use the Pillow library if you are using python).
Here is the structure of each glyph in the font file:

Shift-JIS encoding of that glyph (2 bytes) (Note: the game uses binary search to find the right glyph, so this encoding value should always be in increasing order.)
0x00 (4 bytes)
width of the glyph (1 byte)
height of the glyph (1 byte)
width of the border glyph (1 byte) (a larger bitmap used to draw borders, you can use the same glyph if you don't need border)
height of the border glyph (1 byte)
the glyph (a width * height grayscale bitmap)
the border glyph (a (width of the border glyph) * (height of the border glyph) grayscale bitmap)

Here's the source code of to_font, which may looks confusing too...

Code: Select all

def to_font(ch, fontpath, height):
    enc = ENCODE_TABLE[ch]
    uni_idx = ord(ch)
    ret = bytearray()

    # read font bitmap
    fin = open(fontpath + '/' + str(uni_idx) + '.txt', 'rb')    # I generated the bitmaps and stored them to a text file.
    img = fin.read()
    mc = len(img)
    fin.close()

    metrics = FONT_METRIC

    width = metrics[uni_idx][4]

    if len(enc) == 1:
        L = [str(enc[0]), '0x0', '0x0', '0x0', '0x0', '0x0', str(width), str(height), str(width + 2), str(height + 2)]
    else:
        L = [str(enc[1]), str(enc[0]), '0x0', '0x0', '0x0', '0x0', str(width), str(height), str(width + 2), str(height + 2)]
    ret.extend(bytes(int(x, 0) for x in L))

    canvas = bytearray(width * height)
    # I removed some codes here because I did some tricks to draw the image onto the canvas, which may look confusing.
    # (I need to do so because my language has some special feature.)
    # You can use 'img' directly without using the canvas
    # e.g.
    # ret.extend(bytes(img))
    # ret.extend(bytes(img))
    # return ret
    
    ret.extend(bytes(canvas))
    ret.extend(bytes(to_border(img, mc, width, height, metrics, uni_idx)))
    return ret

[zkdldk95]

The font file uses 8-bit grayscale bitmaps.
You can generate grayscale bitmaps from ttf/otf fonts using any font engine. For me I use freetype. (You can use the Pillow library if you are using python).
Here is the structure of each glyph in the font file:

Shift-JIS encoding of that glyph (2 bytes) (Note: the game uses binary search to find the right glyph, so this encoding value should always be in increasing order.)
0x00 (4 bytes)
width of the glyph (1 byte)
height of the glyph (1 byte)
width of the border glyph (1 byte) (a larger bitmap used to draw borders, you can use the same glyph if you don't need border)
height of the border glyph (1 byte)
the glyph (a width * height grayscale bitmap)
the border glyph (a (width of the border glyph) * (height of the border glyph) grayscale bitmap)

Could you see my modified font file and check what's wrong?

At the end of the file, i added the glyph '가' (at 0x2a834)

And then i changed offset tables (head and bottom both) and
number of characters (84 -> 85 at 0x2b18c) and
the data size(variable-length quantity).

But still the game is crashed.

it is English-font file, but you might understand it.

[erito]

Hey, I know this post have been inactive for quite some time, but I just wanted to ask a little thing:

...
84 A8 70 -> 0x11470 next 0x11470 bytes (to the start offset of text index table)
...

I might be really dumb, but what is the relation between "84 A8 70" and "0x11470", how do you convert between the two?
Thanks!

EDIT: Forgot to check on variable-length quantity, my bad.
Quick brief for anyone passing by, using the example of `00009ed8.sir`:

• Take the offset of the TOC (`0x2A834`)
• Convert it to binary (`10 1010 1000 0011 0100`)
• Group them by 7, starting from the right. If the left group contains less than 7 character, add 0s : (`0001010`, `1010000`, `0110100`)
• Add a 1 at the start of each group, except the right one, where you must add a 0 : (`10001010`, `11010000`, `00110100`)
• Combine it all ! (`100010101101000000110100`)
• Convert it in hex. (`0x8AD034`)
• Put it after `04 08 80` at the end of the file
• Note: While most of the offset are written in little endian, this one seems to be written in big endian (which mean that the bytes aren't reversed)

[erito]

Hey, here I am again, always one year late...
So I ran into the same problem as zkdldk95: added a character for my language ("é", identified in Unicode by U+009E). So I tried, but each times it fails, even though I checked many times that I have the correct offsets, VLQ, etc...
I tried to go in a more in-depth experiment, trying to change each components of the file individually to understand what might cause crashes. Judging by my experiment, it seems like it could be:

• Beginning offset of table, however already checked that carefully
• The VLQ, but I also checked that

I'm really wondering on what could be wrong right now... At this point, I'm really wondering if reimport2, which was upgraded since your last posts, really does the job for this particular case? Do anyone who managed to actually add characters in the font files without crashes could explain how they did?
Looking forward to it, thanks!

[erito]

OK so I actually tried to rewrite the whole file, the characters are messy but that's no big deal...
The game crashed when clicking the start button; now, it just won't display the character I created (except the first one)... Why does only the first one works (°) and not the second (À) and the following? Do I have to, like, escape the other characters? I've noticed that " ' " are actually written as " ,r " in the text files, so is there this kind of manipulation for that too?
(of course, I changed `00009f0e.sir` in order to make those characters appear)

EDIT:
OK so now, just tried some new things, and it seems that when I try to print '84BE' or '84something', it only displays a big D on a white background, although the glyph entry for this wasn't that D at all...

[erito]

Well, I think no one's watching this, but it may be useful for anyone trying to translate this, so here's what I found out:
The best is to rewrite the font file (`00009ed8.sir`) by yourself. It might take some time, but honestly that's the easiest way. Try to have a height of 35 px for each character. To create new characters (just as I want to add é, for instance), you have to modify the encoding of the glyph. é was 0xE900 (little endian), but in order to make it work, I had to transform it to 0xE981. Then, in your text file (for instance, `00009f0e.sir`, the beginning of the game) you just put `81E9` each time you want to use this character. Please, note that if characters are put in the wrong order in `00009ed8.sir`, the character won't appear.