Silmarils compression algorithm on files with io extension (or Ishar compression)

[AleWin32]

A lot of games from Silmarils company use game files with .io extension and those are compressed. All these Silmarils games are full of .io files:

Targhan
Colorado
Crystals of Arborea
StarBlade
Metal Mutant
Ishar I
Ishar II
Ishar III
Transarctica (or Artic Baron)
Bunny Bricks

but there are more for sure, i have not checked them all. So discovering the compression algorithm will give us the possibility to decompress game data of a lot of Silmarils games.
The .io files store different kind of data: music, graphics, texts, etc.

Looks like a common file on all of those games is MAIN.IO, also BLANCPC.IO, sure there are others, i have not checked all.

In particular i have investigated Ishar: Legend of the fortress.

GAME DOWNLOAD (french version)
You can download the game from here (if you want to run tests and to investigate executing the game):
https://16bit.pl/download/games/pc/isharlegendofthefortress.zip

NOTE: Use the numeric pad of the keyboard to select the language when prompted.
NOTE2: The spanish version (not the one i posted up here) does not have the language selection screen! The file that displays that is MAIN.IO, so if you take the MAIN.IO from the french version and overwrite it on the spanish, you will get that screen.

COMPARING FRENCH WITH SPANISH VERSION
All files are identical across spanish and original version except:

MAIN.IO (the one from french shows a language selection screen, spanish don't)
MESSAGE.IO (spanish version has this in spanish, and french version in french)
MESSAGED.IO
MESSAGEE.IO
SOS.IO
START.EXE (French version has some little differences like fading out the music on title screen when space bar is pressed, spanish version stops it abruptly)
TEXTIN.IO

I used WinMerge to compare across french and spanish version files:



ABOUT THE GAME EXECUTABLE (PACKED) START.EXE

The exe from Ishar is packed with PACK201 (PACK 2.01). I detected it using SH Archive Identifier (SHAID, Archid):


Then i checked it with PACK 2.01 itself and it detect that it is packed with it:


Them, i used "UUP v1.4 - Universal exe-file UnPacker" with the exe and successfully extracted it (and works if you execute it!):


The packed start.exe file i used for the screenshots are from the Spanish version.

UNPACKED GAME EXECUTABLES

Spanish unpacked START.EXE:
Download:
https://mega.nz/#!xdcwhI5L!jYOnoU1G6Ob9 ... BFHAuPpiyw
Packed EXE MD5:
1CC16F2C49C6CE5751893C02632443C4

French unpacked START.EXE:
Download:
https://mega.nz/#!IZEzhYbB!E-jmfGyfKyWz ... v3g0Ztvd78
Packed EXE MD5:
DE1B671638EC5EC9FC93E2304CE1DF44


Size difference between unpacked EXE versions:
Spanish: 75 KB
French: 88 KB

A note when unpacking the French EXE version is that i get this message (that was not displayed on spanish EXE version) but the unpacked EXE works:


Now we can dissasembly the executable to do reverse engineering, unfortunately i don't know very much x86 assembler.

HOW TO CHECK WHAT FILES ARE USED ON EACH MOMENT
I run the game on Dosbox and i use the freeware tool ProcessActivityView over the Dosbox process to see what files the game access on every moment. You can also use ProcessMonitor that gives even more details: all the offsets on each file the game is accessing.



EXAMPLES OF TWO FULLSCREEN IMAGE FILES

These are good to investigate because we know they have the same kind of data: two images of 320x200 (64.000 pixels), so we can compare each other and take a look at the header bytes to see differences, but unfortunately i have not found what seems to be the bytes that indicate the uncompressed data size.

- Starting the game, after some scenes (Silmarils logo and others) you will see a fullscreen image (320x200) of the game credits, the file used for this is AUTEUR.IO, it's like that:


Copy and pasting random little blocks of bytes from the file itself (AFTER what it looks like it is the header) over itself you can change the background of the image when the game displays it:


As you can see the background has been modified but not the red characters, but if you continue modifying blocks of bytes of the file, you will also modify the red characters (look on the bottom):


So looks like they are stored in two planes.

- When the player is dead, another fullscreen of 320x200 image will be shown:


SWAPING THE IMAGES ON THE GAME
OK. So now, the classic trick... What if we copy AUTEUR.IO file over DEAD.IO? Now supposedly, the credits must be shown when the player is dead but... you get a black screen (although music plays) and you can continue the game when pressing space. But I discovered how to get the image shown!

MAKING THE FULLSCREEN IMAGE SWAP WORK
Comparing the first bytes of AUTEUR.IO and DEAD.IO you will see some little differences on the first bytes.

Looks like the byte on offset 16 (starting by 1 on the file) identifies something like the scene:

AUTEUR.IO has 0x98 value

DEAD.IO has 0xC0 value

So writing now the value of 0xC0 (the value that the dead scene file must have) over the 0x98 value to it works and now it displays the AUTEUR image when you're dead.

You can try my modified AUTEUR.IO file to see the dead screen instead of the credits screen (is a modified DEAD.IO file: it is the same except the byte at offset 16 that is taken from AUTEUR.IO to match the value that the game expects, because if it does not match, the image is not shown and you will see a black screen instead):
https://mega.nz/#!MJdnnCZZ!WCzIjDvYxaWn1DaFpfwOqDIZQwjG5WYYYOeoiLv5JOU

STRINGS
The strings are stored on those files. On the french version of the game:

MESSAGE.IO: strings in french

MESSAGED.IO: strings in german

MESSAGEE.IO: strings in english

MESSAGEI.IO: strings in italian

this is an advantage because they store the same strings but for different languages, so we can compare those files to examine how the bytes on the header are different because the file size differs, but i could not get where the uncompressed size bytes are on those MESSAGEX.io files

On the french MESSAGE.IO file, i found 4 uncompressed strings:

• "CARAC" (in characteristics panel in the game, "CARAC" is used to print the first part of the string "CARACTERISTIQUES :" and the first character "C" to print the first character of "CONSTITUTION : ")
• "TU" (in characteristics panel in the game, "TU" is used to print the characters 7 and 8 in the string "CONSTITUTION : ")
• "PHYS" (used to print "SCHLOUMZ : REGENERATION PHYSIQUE". This is a supposition, i have not checked this because you have to arrive to some part of the game to see that string. In ARAMIR warrior panel there is a string with "PHYSIQUE : " but it does not change if you change "PHYS" to another characters, so looks like it is used to print "SCHLOUMZ : REGENERATION PHYSIQUE")
• "INVUL" (there are 2 strings that contain this string: "CLOPATOS : INVULNERABILITE" and "INVULNERABILITE 6" but i have not checked in the game if they change if i modify "INVUL")

You can change those strings with anothers characters, for example, you can change "CARAC" by "ABCDE" and you will see the changes in the game where they are used.

The "CARAC" and "TU" strings on MESSAGE.IO:


Used to print some word characters on characteristics panel:


Now, if you change "CARAC" and "TU" to "ABCDE" and "FG":


you will see the changes:


Also, "CARAC" string is used to print the competences panel:


so having "CARAC" changed to "ABCDE" you will see the changes:


Also, here are the "PHYS" and "INVUL" strings but i have not checked where they appear on the game:


So, the conclusion relative to compression is that MESSAGE.IO file:

• has some uncompressed string parts: "CARAC", "TU", "PHYS" and "INVUL" (maybe there are more)
• has the most string parts compressed (there are not so many kind of "valid" strings or dictionary words)
• the most important, it use some kind of compression that characters are pointed from other strings to reuse them, as you can see in my examples: "CARAC" is the source string for the ingame strings "CARACTERISTIQUES", "ARMES 1 MAIN", "ARMES 2 MAINS", "ARMES DE JET" AND "COMPETENCES".

I have no compression algorithm knowledge but maybe some of you can help. As i told, discovering the Silmarils compression algorithm would make us having the possibility to decompress lot of game data from the dozens other games they have.

UNCOMPRESSED MESSAGE.IO (french and spanish)

I successfully extracted uncompressed MESSAGE.IO (french and spanish) from memory using the DOSBox debugger, you can download it here:
https://mega.nz/#!1RdDgKZA!R0zED1WQA641WsqLCUGxSFOIS7L5RdhmMxEPyngTfjA

These files are what was written in memory when the file MESSAGE.IO was read, i carefuly executed it step by step with a breakpoint where the destination byte was written.

Anyone want to help?

[aluigi]

You forgot the most important thing: the files.

[AleWin32]

You forgot the most important thing: the files.

Updated on my first post.

[AleWin32]

Game executable investigation added on my first post

[AleWin32]

MESSAGE.IO file (french and spanish) uncompressed extracted from a memory dump on my first post, so now we have the compressed and the uncompressed file, but we don't know the compression algorithm...

[Zeloran]

While searching for a way to decompress Silmarils data files, I found this forum and want to share my findings.

It seems like the compression algorithm has already been found years ago. On eab there is a very old thread where extracting music files from Silmarils Amiga games is discussed. The interesting bit from this thread was a statement from a user named meynaf, who shared an unpacker (decruncher) for Silmarils games on the Amiga:

Note that unlike WT's one, my xfd decruncher produces files that can still be used by the game engine.

I managed to find his file on the site’s ftp server and ran a few tests

I used the file accueil.*o (the main menu file) from the game Ishar 3. The versions I used are:

ACCUEIL.AO (25kb) - Atari ST german
ACCUEIL.CO (25kb) - Amiga OCS french
ACCUEIL.DO (75kb) - Amiga AGA multilingual (english, french, german)
ACCUEIL.FO (75kb) - Macintosh / Atari Falcon multilingual (english, french, german)
ACCUEIL.IO (75kb) - DOS spanish

All files are compressed. The size difference between the Atari ST/ Amiga OCS versions and the rest is simply the lower color palette and resolution of the first two versions. The file ACCUEIL contains the texts and the graphics of the main menu.

Atari ST

Amiga AGA


Testing Environment

First of all, let me share my testing environment, so you can play with it yourself.:

https://mega.nz/#!9Soj1QqI!MOVm9bHGAhVWbvPabtq7zpvm_JvMYe6CZISkldaaH8c

In the Amiga Folder there is a batch file named workbench.bat which will start FS-UAE with a preinstalled Workbench environment. The folder .\Hard Drives\Playground is mounted as HD, so we can easily transfer files from or to the Amiga, The decruncher from the eab thread is already installed in workbench.

The second batch file in the amiga folder is named Ishar3.bat and will boot the game directly from the Folder .\Hard Drives\Ishar3

But for now, let’s start with the workbench.
In the folder Project Silmarils\Data\Multiformat Comparison\Untouched Files I have collected some data files from different Ishar 3 platform releases. We copy all files to our Playground folder and boot up the workbench.

Test #1 - accueil.do

Let’s start with accueil.do
In a shell window we type
Playground:
Xfddecrunch accueil.do accueil2.do



The extraction is successful and we can leave the Amiga for now (Alt+F4)
The new accueil2.do has a size of 129kb.
We open it in a hex editor

Et voila. The texts of the main menu. But will the game run with the uncompressed file? Let’s find out.
We rename the file back to accueil.do and copy it to /Hard Drives/Ishar3

Now we start ishar3.bat and…

The game works
So it safe to assume that Silmarils games can indeed run no matter if the data is compressed or extracted. At least on the Amiga.
But what about the other platforms?

Test #2 - accueil.ao, .co & .fo

We now try to extract the Amiga OCS, Atari St & Macintosh/falcon files.
All files are recognized by the decruncher and are extracted without problems.

These are extracted create.*o (character creation menu) files, but accueil works as well


I havent tested extracted .ao & .fo yet, so I don't know, if the game will run.
I have tested the Amiga OCS/AGA files and they work fine.

Test #3 - accueil.io DOS

Now for the DOS files (.io)
The decruncher fails to recognize them as compressed so we cannot extract them. Or can we?



If we modify the dos files by copying the first few bytes of the Amiga AGA files, the decruncher recognizes and extracts them. By opening them in a hex editor we can confirm, that the extraction was successful.






But will the file work?
We can rename and copy it to the DOS/Ishar3 folder and start dosbox.
The game crashes with an error code.

So I assume that the data itself was extracted correctly, but the file informations/headers are incorrect. Or maybe the DOS version doesn’t work with uncompressed files.

Test #4 - Does the DOS version work with uncompressed files?

And now for the reason I used the spanish version of Ishar3 in Dos.
The floppy version comes with a compressed accueil.io file (75kb), but we have access to an uncompressed version.
The Amiga AGA version has 3 different languages: english, french and german. The file accueil.do contains the texts in all 3 languages.
But the DOS CD-Rom release added a 4th language: spanish. The original accueil.io still only contains the 3 languages, but a second file named accueils.io contains the spanish texts only. And by looking at the file size (109kb) and in a hex editor we can see that this file is uncompressed.

I have added both files to /Data/Ishar 3 Dos and renamed accueils.io to accueil.io
If we copy the uncompressed accueil.io to DOS/Ishar3 and start the game, it will boot just fine.

Conclusion:

We have an extraction tool that will give us uncompressed, usable files for the Amiga version of Silmarils games.
Atari and Macintosh extraction seems to work but resulting files are untested for now.
PC extraction only works with modified files, but uncompressed files are unusable. Our uncompressed file has a size of 129kb, the original uncompressed file that works with the game has a size of 109kb
We have an original compressed and an uncompressed accueil.io that can be compared.

And the most important thing: The amiga decruncher library comes with the source code, so someone with programming skills should be able to use it for further examination.
I have absolutely no programming knowledge, so i can't do anything with the source code.

Code: Select all

; silmarils file decruncher, basé sur le mien et celui de wanted team

marge equ 256         ; si la décompression déborde du buffer

 moveq #-1,d0
 rts
 dc.b "XFDF"         ; xfd id
 dc.w 1            ; version
 dc.w 0            ; reserved
 dc.l 0,0         ; private
 dc.l slave1         ; first slave

 dc.b "$VER: XFD Silmarils file decruncher 1.01 (17.11.2002)",0

name1 dc.b "Old Silmarils",0
name2 dc.b "Old Silmarils Main",0
name3 dc.b "Silmarils",0
name4 dc.b "Silmarils Main",0
name5 dc.b "Old Silmarils (interlaced)",0
name6 dc.b "Old Silmarils Main (interlaced)",0
 even

; macro declare slave : n°, next, minsize
declslv macro
slave\1
 ifne \2
 dc.l slave\2      ; next
 else
 dc.l 0         ; dernier
 endc
 dc.w 2         ; version
 dc.w 39      ; xfdmaster version
 dc.l name\1      ; format name
 dc.w $304      ; flags
 dc.w 0         ; maxlen special info
 dc.l check\1      ; check routine
 dc.l decr\1      ; decrunch routine
 dc.l 0         ; recog segment
 dc.l 0         ; decrunch segment
 dc.w 0,0      ; slave/replace id
 dc.l \3      ; min size (wanted team met 14)
 endm

 declslv 1,2,8
 declslv 2,3,$18
 declslv 3,4,$10
 declslv 4,5,$20
 declslv 5,6,8
 declslv 6,0,$18

; note : bug dans les premiers xfd silmarils decrunchers : NetBSD executables
; -> je peux pas tester...

; check (main avec b0=0 car 4(a0) sera aussi 0, donc c'est égal)
check1 moveq #1,d1         ; old
 bra.s check
check2 moveq #0,d1         ; old main
 bra.s check
check3 moveq #3,d1         ; new
 bra.s check
check4 moveq #2,d1         ; new main
 bra.s check
check5 moveq #5,d1         ; old (interlace)
 bra.s check
check6 moveq #4,d1         ; old main (interlace)
check
 sub.l (a0),d0            ; d0=taille24bits : crunch-norm
 btst #23,d0            ; signe du résultat
 beq.s .oups            ; taille crunchée > taille normale ?!
 move.b (a0),d0
 andi.b #$9f,d0            ; 81(old) ou a1(new)
 cmpi.b #$81,d0            ; (wt accepte aussi 80/C0)
 bne.s .oups
 move.w 4(a0),d0         ; 0000 main.co 0001 data.co
 eor.b d0,d1            ; bits 0 identiques -> b0 d1 à 0
 andi.w #$fffe,d0         ; le reste doit être 0
 bne.s .oups
 move.b (a0),d0            ; bit 5 nous intéresse (old/new)
 andi.b #$20,d0
 lsr.b #4,d0            ; -> en bit 1
 eor.b d0,d1            ; bits 1 identiques -> b1 d1 à 0
 move.b (a0),d0
 andi.b #$40,d0            ; bit 6, cette fois
 lsr.b #4,d0            ; -> en bit 2
 eor.b d0,d1
 bne.s .oups            ; (donc d1 totalement à 0)
 move.l (a0),d0
 andi.l #$ffffff,d0
 cmpi.l #$100000,d0         ; jamais >1m (512k) dans jeux silmarils
 ble.s .ok            ; (record : montagne.do, 241k-ishar3)
.oups moveq #0,d0         ; pas bon
 rts
.ok
 move.l d0,4(a1)         ; taille saved data
 add.l #marge,d0
 move.l d0,(a1)
 moveq #1,d0            ; indiquer ok
 rts


; decrunch routines
decr1 moveq #0,d1
 bra.s decr
decr2 moveq #1,d1
 bra.s decr
decr3 moveq #2,d1
 bra.s decr
decr4 moveq #3,d1
 bra.s decr
decr5 moveq #4,d1
 bra.s decr
decr6 moveq #5,d1
 bra.s decr
decr
 movem.l d2-d7/a2-a6,-(a7)   ; si lui le fait, y a une raison
 move.l $3c(a0),a1      ; outbuf
 move.l a1,a2
 move.l 4(a0),d6      ; src size
 move.l (a0),a0         ; src addr
 move.l (a0),d7
 andi.l #$ffffff,d7      ; dest size
 addq.l #1,a0
 moveq #4,d0         ; norm header (6 bytes)
 btst #0,d1
 beq.s .norm
 moveq #$14,d0         ; main header ($16 bytes)
.norm move.b #1,(a1)+
.loop move.b (a0)+,(a1)+
 dbf d0,.loop
 add.l a2,d7         ; -> end of dest buff
 sub.l a1,d7         ; -> reste de dest à remplir
 btst #1,d1
 beq.s .old
 bsr newm         ; new method (>=ishar1)
 bra.s .done
.old
 bsr.s oldm         ; old method (<ishar1)
.done
 movem.l (a7)+,d2-d7/a2-a6
 moveq #1,d0
 rts


; decrunch routines : reçoivent a0/d6=src, a1/d7=dest (adr/taille)
; si erreur : compléter avec octets nuls (ça se produit sur certains fichiers)
; si output buffer dépassé, quitter simplement

; ancienne méthode
oldm
 lea (a0,d6.l),a2
 lea (a1,d7.l),a3
 moveq #1,d3
 btst #2,d1
 beq.s .hmpf
 moveq #8,d3
.hmpf move.l d3,d4
.loop0 cmpa.l a1,a3
 ble.s .fin
 bsr.s .in
 move.b d1,d2
 bclr #7,d2
 bne.s .rep
.loop1 bsr.s .in
 bsr.s .out
 subq.b #1,d2
 bne.s .loop1
 bra.s .loop0
.rep bsr.s .in
.loop2 bsr.s .out
 subq.b #1,d2
 bne.s .loop2
 bra.s .loop0
.in cmpa.l a0,a2
 blt.s .zero
 move.b (a0)+,d1
 rts
.zero moveq #0,d1
 rts
.out move.b d1,(a1)
 add.l d3,a1
 rts
.fin sub.l d7,a1
 addq.l #1,a1
 subq.l #1,d4
 bne.s .loop0
 rts

; nouvelle méthode
newm
 lea (a0,d6.l),a4      ; end of input buffer
 lea (a1,d7.l),a2      ; end of output buffer
; code identique à partir de là
 subq.l #8,a7
 move.l (a0)+,(a7)
 move.l (a0)+,4(a7)
 clr.w d7
.loop cmpa.l a2,a1
 bgt.s .exit
 moveq #1,d0
 bsr.s .vache
 tst.b d5
 beq.s .momie
 moveq #0,d2
.zombie moveq #2,d0
 bsr.s .vache
 add.w d5,d2
 cmp.w #3,d5
 beq.s .zombie
.qqch moveq #8,d0
 bsr.s .vache
 move.b d5,(a1)+
 dbf d2,.qqch
 cmpa.l a2,a1
 bgt.s .exit
.momie bsr.s .bouh
 clr.w d0
 move.b 0(a7,d5.w),d0
 andi.w #3,d5
 beq .seins
 move.w d5,d2
 bsr.s .vache
.femelle neg.w d5
.sexe move.b -1(a1,d5.w),(a1)+
 dbf d2,.sexe
 bra.s .loop
.seins bsr .vache
 move.w d5,d3
 clr.w d2
.cervelle bsr.s .bouh
 add.w d5,d2
 cmp.w #7,d5
 beq.s .cervelle
 move.w d3,d5
 addq.w #4,d2
 bra.s .femelle
.bouh moveq #3,d0
.vache sub.b d0,d7
 bmi.s .worm
 clr.w d5
 rol.l d0,d5
 rts
.worm add.b d0,d7
 clr.w d5
 rol.l d7,d5
 swap d5
 cmpa.l a4,a0      ; ces deux lignes sont un ajout à moi
 bge.s .argh      ;
 move.w (a0)+,d5
 swap d5
 sub.b d7,d0
 moveq #$10,d7
 rol.l d0,d5
 sub.b d0,d7
 rts
; tout ce qui suit est ajouté par moi
.argh
 addq.l #4,a7
.zero clr.b (a1)+
 cmpa.l a1,a2
 bhi.s .zero
.exit
 addq.l #8,a7
 rts

[dim568]

For DOS version

Bytes 00-01 ~ H0
Bytes 02-03 ~ H1
Bytes 04-05 ~ H2 - ExtraHeader(Read next 16 bytes)
HeaderSize = 6 or 22(6 basic and 16 extra)
Unpacked size = SWAP16(H1&0xFF)*16+(H0-HeaderSize)
First logic switch (upper bit of H1), 0-uncompressed 1-compressed
If file compressed next logic switch is upper 7 bits of H1, program compare it with 0xA0 (maybe just check bit 13 of H1, because bit 15 checked on previous step and =1)

[VadimK]

Sorry for bumping old thread, but this looks like the only place on net for anyone messing with Silmarils games.

I tried to extract some resources and find some useful info. Tested on Amiga AGA versions of Ishar 2 and 3. Should work on Atari and Mac versions as well.

Strings are prefixed by two bytes 0x1E, 0x04 and are null terminated. Unexpectedly in Ishar 2, it looks like there aren’t any false positives. Not tested elsewhere.
Palettes are prefixed with 0xFE, 0xFF, 0x00, 0x00 and for 256 color versions of Ishar its 256 * 3 bytes in length. Atari ST and Amiga OCS versions might have only 32 color palette (0-15 UI 16-31 game view) but I haven't tested it yet.
Sprites are stored as chunky bitmaps either 4 or 8 bit. Each bitmap is prefixed with some basic data:

int8 type; // values 0x10 = 4bit, 0x14 and 0x16 8bit
int8 unknown0;
int16 width; //depending on game platform and host cpu, you might have to swap bytes to get correct values, also dimensions are calculated from zero, so fullscreen is 319x199)
int16 height;
int16 unknown1;

bitmap width * height ( / 2 for 4bit)

Just searching for values 0x10, 0x14, 0x16 reading width, height data and then testing if it contains sane values is enough to read all sprites with just a few false positives.
For 4bit UI sprites palette indexes are 0-15 for game view its 16-31.

Thats it for now.

[VadimK]

Ok, some more info. Finding palette using aforementioned method in Ishar 3 results mostly in getting garbage.

Maps are stored in 'cont?.fic' files, it can be easily visualized as 8bit bitmap with width 60px for Ishar 2 and 57px for Ishar 3.

cont3.png