LEGO Universe - .pk & sd0 (UPDATED)
-
ProfBrickkeeper
- Posts: 13
- Joined: Mon Jun 13, 2016 7:52 pm
LEGO Universe - .pk & sd0 (UPDATED)
Recently, some friends and I have been working with modding the MMOG LEGO Universe and its files as part of private server efforts, but have found that some of the pack (.pk) files seem to contain certain files which cannot be properly extracted using our current tools. (See this chart of failed files in the included .pk file) The good news is that the files are not compressed nor encrypted inside the pack files, which makes for easier extraction and reading of them. I have included the file shown in the previous chart (forkpefx.pk) and was curious whether anyone here would be interested in taking a swing at writing a script to properly extract the files. Thanks!
Last edited by ProfBrickkeeper on Mon Jul 25, 2016 11:08 pm, edited 3 times in total.
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk
Yeah there is a table with information about the stored files but no filenames:
http://aluigi.org/bms/lego_universe_pk.bms
http://aluigi.org/bms/lego_universe_pk.bms
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk
The file names and paths they should go in are stored in some other txt files, along with MD5s and such that are used to identify them in the PKs:
https://www.dropbox.com/s/r4bs34c3mfvph ... s.zip?dl=0
The main files of note in there are trunk.txt (contains all file paths/names, their hashes, and sizes), and maybe primari.pki (not sure what exactly that's for or if it'd be needed for extracting the PKs).
Also, it looks like some of the files within ARE compressed. The format seems to be "sd0" but I'm not sure if it's proprietary or not. When the client would patch, all the files were downloaded compressed as .sd0 files, but only some of them remain that way in the PKs.
So there's two parts to this:
A) Extracting the files from the PKs to their proper file paths/with their correct names, using the info in the versions folder
B) Decompressing the ones that are compressed as sd0 files
I'm currently gathering some examples of sd0 files to compare with their uncompressed counterparts - I'll edit this post with more once I've got them...
Edit: Alright, here's some stuff.
First, some more examples of .pk files:
https://www.dropbox.com/s/jzyoxictmyx9i ... s.zip?dl=0
front6.pk and maps2.pk contain some sd0-compressed files, tex3.pk does not.
And some sd0-compressed files and non-compressed versions to compare with:
https://www.dropbox.com/s/jvl4ggfi96kew ... n.zip?dl=0
As said in the note included, cdclient.fdb can be found in front6.pk, and the nimbus_station files can be found in maps2.pk.
Also, this person seems to have worked out at least some of how sd0 works, if not all (see decompress_sd0.py):
https://bitbucket.org/lcdr/utils/src/
I haven't gotten it it to work yet myself, but assuming it's correct, should save a good amount of work. I'll toy with it a bit more now. (Edit - Nope, still no go on that script... Though it's likely just a problem with me not knowing how to use it. Of course people have decompressed sd0s before, but there's no easy solution for handling lots at once yet... It'd be helpful if it could just be taken care of at the same time the files are extracted from the PKs.)
https://www.dropbox.com/s/r4bs34c3mfvph ... s.zip?dl=0
The main files of note in there are trunk.txt (contains all file paths/names, their hashes, and sizes), and maybe primari.pki (not sure what exactly that's for or if it'd be needed for extracting the PKs).
Also, it looks like some of the files within ARE compressed. The format seems to be "sd0" but I'm not sure if it's proprietary or not. When the client would patch, all the files were downloaded compressed as .sd0 files, but only some of them remain that way in the PKs.
So there's two parts to this:
A) Extracting the files from the PKs to their proper file paths/with their correct names, using the info in the versions folder
B) Decompressing the ones that are compressed as sd0 files
I'm currently gathering some examples of sd0 files to compare with their uncompressed counterparts - I'll edit this post with more once I've got them...
Edit: Alright, here's some stuff.
First, some more examples of .pk files:
https://www.dropbox.com/s/jzyoxictmyx9i ... s.zip?dl=0
front6.pk and maps2.pk contain some sd0-compressed files, tex3.pk does not.
And some sd0-compressed files and non-compressed versions to compare with:
https://www.dropbox.com/s/jvl4ggfi96kew ... n.zip?dl=0
As said in the note included, cdclient.fdb can be found in front6.pk, and the nimbus_station files can be found in maps2.pk.
Also, this person seems to have worked out at least some of how sd0 works, if not all (see decompress_sd0.py):
https://bitbucket.org/lcdr/utils/src/
I haven't gotten it it to work yet myself, but assuming it's correct, should save a good amount of work. I'll toy with it a bit more now. (Edit - Nope, still no go on that script... Though it's likely just a problem with me not knowing how to use it. Of course people have decompressed sd0s before, but there's no easy solution for handling lots at once yet... It'd be helpful if it could just be taken care of at the same time the files are extracted from the PKs.)
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Do you have a directory tree?
I mean the list of the files at least to understand where the PK files are located compared to versions/*.txt becaus I need to instruct quickbms to open the txt first.
sd0 are easy to handle, just chunked files, going to add them to the script automatically.
I mean the list of the files at least to understand where the PK files are located compared to versions/*.txt becaus I need to instruct quickbms to open the txt first.
sd0 are easy to handle, just chunked files, going to add them to the script automatically.
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Ah ok, I found the whole tree.
Problem solved
I'm working on the script.
Problem solved
I'm working on the script.
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Ah man, you're awesome. 
I know you said you've got it, but I just noticed that the txt files list quite a few .pk files that don't actually exist, mostly (entirely?) related to localizations of the game that never actually happened. So just to clarify - here's a tree showing all the .pk files that are truly in the game (also where everything is relative to the versions folder). Note that there's only two localized pk files, for en_gb, near the bottom of the list:
I probably didn't need to include the "LEGO Universe" bit as that's where the game was installed to, but oh well.
So, be aware the .txt files might possibly be listing more files that don't actually exist.
I know you said you've got it, but I just noticed that the txt files list quite a few .pk files that don't actually exist, mostly (entirely?) related to localizations of the game that never actually happened. So just to clarify - here's a tree showing all the .pk files that are truly in the game (also where everything is relative to the versions folder). Note that there's only two localized pk files, for en_gb, near the bottom of the list:
Code: Select all
LEGO Universe\client\res\pack\anim1.pk
LEGO Universe\client\res\pack\anim2_1.pk
LEGO Universe\client\res\pack\anim2_2.pk
LEGO Universe\client\res\pack\anim2_3.pk
LEGO Universe\client\res\pack\anim2_4.pk
LEGO Universe\client\res\pack\anim2_5.pk
LEGO Universe\client\res\pack\anim3.pk
LEGO Universe\client\res\pack\anim4.pk
LEGO Universe\client\res\pack\animx.pk
LEGO Universe\client\res\pack\brickm1.pk
LEGO Universe\client\res\pack\brickmx.pk
LEGO Universe\client\res\pack\brickp1.pk
LEGO Universe\client\res\pack\brickp2.pk
LEGO Universe\client\res\pack\brickpx.pk
LEGO Universe\client\res\pack\fontx.pk
LEGO Universe\client\res\pack\forkpef1.pk
LEGO Universe\client\res\pack\forkpef2.pk
LEGO Universe\client\res\pack\forkpefx.pk
LEGO Universe\client\res\pack\forkpx.pk
LEGO Universe\client\res\pack\front1.pk
LEGO Universe\client\res\pack\front2_1.pk
LEGO Universe\client\res\pack\front2_2.pk
LEGO Universe\client\res\pack\front2_3.pk
LEGO Universe\client\res\pack\front2_4.pk
LEGO Universe\client\res\pack\front2_5.pk
LEGO Universe\client\res\pack\front2_6.pk
LEGO Universe\client\res\pack\front3_1.pk
LEGO Universe\client\res\pack\front3_2.pk
LEGO Universe\client\res\pack\front3_3.pk
LEGO Universe\client\res\pack\front4.pk
LEGO Universe\client\res\pack\front6.pk
LEGO Universe\client\res\pack\front_bricks.pk
LEGO Universe\client\res\pack\front_maps.pk
LEGO Universe\client\res\pack\macrox.pk
LEGO Universe\client\res\pack\maps1.pk
LEGO Universe\client\res\pack\maps2.pk
LEGO Universe\client\res\pack\maps4.pk
LEGO Universe\client\res\pack\maps5.pk
LEGO Universe\client\res\pack\mapsx.pk
LEGO Universe\client\res\pack\mesh1.pk
LEGO Universe\client\res\pack\mesh2.pk
LEGO Universe\client\res\pack\mesh3.pk
LEGO Universe\client\res\pack\mesh4.pk
LEGO Universe\client\res\pack\meshb1.pk
LEGO Universe\client\res\pack\meshb2.pk
LEGO Universe\client\res\pack\meshb3.pk
LEGO Universe\client\res\pack\meshb4.pk
LEGO Universe\client\res\pack\meshenv2a.pk
LEGO Universe\client\res\pack\meshenv2b.pk
LEGO Universe\client\res\pack\meshenv2c.pk
LEGO Universe\client\res\pack\meshenv3a.pk
LEGO Universe\client\res\pack\meshenv3b.pk
LEGO Universe\client\res\pack\meshenv3c.pk
LEGO Universe\client\res\pack\meshenv3d.pk
LEGO Universe\client\res\pack\meshenv4a.pk
LEGO Universe\client\res\pack\meshenv4b.pk
LEGO Universe\client\res\pack\meshenv5.pk
LEGO Universe\client\res\pack\meshenv6.pk
LEGO Universe\client\res\pack\meshenvsub1.pk
LEGO Universe\client\res\pack\meshenvsub10.pk
LEGO Universe\client\res\pack\meshenvsub11.pk
LEGO Universe\client\res\pack\meshenvsub13.pk
LEGO Universe\client\res\pack\meshenvsub2.pk
LEGO Universe\client\res\pack\meshenvsub3.pk
LEGO Universe\client\res\pack\meshenvsub4.pk
LEGO Universe\client\res\pack\meshenvsub5.pk
LEGO Universe\client\res\pack\meshenvsub6.pk
LEGO Universe\client\res\pack\meshenvsub7.pk
LEGO Universe\client\res\pack\meshenvsub8.pk
LEGO Universe\client\res\pack\meshenvsub9.pk
LEGO Universe\client\res\pack\meshenvsubx.pk
LEGO Universe\client\res\pack\meshenvt.pk
LEGO Universe\client\res\pack\meshenvx.pk
LEGO Universe\client\res\pack\meshlup1.pk
LEGO Universe\client\res\pack\meshlup2.pk
LEGO Universe\client\res\pack\meshlup3.pk
LEGO Universe\client\res\pack\meshlup4.pk
LEGO Universe\client\res\pack\meshlupx.pk
LEGO Universe\client\res\pack\meshx.pk
LEGO Universe\client\res\pack\physics1.pk
LEGO Universe\client\res\pack\physicsenv1.pk
LEGO Universe\client\res\pack\physicsenvx.pk
LEGO Universe\client\res\pack\physicsx.pk
LEGO Universe\client\res\pack\physics_front.pk
LEGO Universe\client\res\pack\scripts.pk
LEGO Universe\client\res\pack\tex1.pk
LEGO Universe\client\res\pack\tex2.pk
LEGO Universe\client\res\pack\tex2x.pk
LEGO Universe\client\res\pack\tex3.pk
LEGO Universe\client\res\pack\tex4.pk
LEGO Universe\client\res\pack\texenv2.pk
LEGO Universe\client\res\pack\texenv3.pk
LEGO Universe\client\res\pack\texenvx.pk
LEGO Universe\client\res\pack\texx.pk
LEGO Universe\client\res\pack\ui1.pk
LEGO Universe\client\res\pack\ui1_front_1.pk
LEGO Universe\client\res\pack\ui1_front_10.pk
LEGO Universe\client\res\pack\ui1_front_11.pk
LEGO Universe\client\res\pack\ui1_front_2.pk
LEGO Universe\client\res\pack\ui1_front_3.pk
LEGO Universe\client\res\pack\ui1_front_4.pk
LEGO Universe\client\res\pack\ui1_front_5.pk
LEGO Universe\client\res\pack\ui1_front_6.pk
LEGO Universe\client\res\pack\ui1_front_7.pk
LEGO Universe\client\res\pack\ui1_front_8.pk
LEGO Universe\client\res\pack\ui1_front_9.pk
LEGO Universe\client\res\pack\uix.pk
LEGO Universe\client\res\pack\_loc\en_gb\front4.pk
LEGO Universe\client\res\pack\_loc\en_gb\mesh1.pk
LEGO Universe\versions\frontend.txt
LEGO Universe\versions\hotfix.txt
LEGO Universe\versions\index.txt
LEGO Universe\versions\primary.pki
LEGO Universe\versions\quickcheck.txt
LEGO Universe\versions\trunk.txt
LEGO Universe\versions\version.txtI probably didn't need to include the "LEGO Universe" bit as that's where the game was installed to, but oh well.
So, be aware the .txt files might possibly be listing more files that don't actually exist.
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Script 0.2 with a new link:
http://aluigi.org/bms/lego_universe.bms
You should just open primary.pki and it will do everything automatically.
Press 'a' when it asks you to overwrite the files, there are many duplicated names probably derived from old versions or old entries.
http://aluigi.org/bms/lego_universe.bms
You should just open primary.pki and it will do everything automatically.
Press 'a' when it asks you to overwrite the files, there are many duplicated names probably derived from old versions or old entries.
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Alrighty, it's getting there!... though, it's only finding 356 files.
Also, 72 files are also getting dumped directly to the output folder with names like 00000049.dds - not sure if that's due to a bug, or them not having entries in trunk.txt.
It's definitely decompressing the sd0 files inside correctly - I fed it front6.pk and it dumped cdclient.fdb properly, albeit as "00000000.dat".
Also, if I use quickbms_4gb_files.exe instead of quickbms.exe, this happens:
What sort of info would be helpful next?
Also, 72 files are also getting dumped directly to the output folder with names like 00000049.dds - not sure if that's due to a bug, or them not having entries in trunk.txt.
It's definitely decompressing the sd0 files inside correctly - I fed it front6.pk and it dumped cdclient.fdb properly, albeit as "00000000.dat".
Also, if I use quickbms_4gb_files.exe instead of quickbms.exe, this happens:
What sort of info would be helpful next?
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
The "356" files displayed there are referred to the last archive, it's not the total number (*edit*, wrong: it's the total amount of files!)
The files with a numeric name are those that are not referenced in trunk.txt, apparently they have been deleted by the developers but their entry continues to exist.
Fixed script 0.2a to work with quickbms_4gb_files.exe, the OFFSET was a 32bit field and not a 64bit one, the 32bit field after it seems the number of deleted files in the archive.
The files with a numeric name are those that are not referenced in trunk.txt, apparently they have been deleted by the developers but their entry continues to exist.
Fixed script 0.2a to work with quickbms_4gb_files.exe, the OFFSET was a 32bit field and not a 64bit one, the 32bit field after it seems the number of deleted files in the archive.
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Huh... Thing is, I'm only getting 356 files extracted. And that's if I tell it to rename files with conflicting names - if I overwrite them instead, I only get 343 extracted files.
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Opssssss, problem found. It was the variable 'i' used by two functions.
Script 0.2.1
Script 0.2.1
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Much better, got 68727 files now... Though there's still some quirks with files ending up in the wrong places. I got 781 files dumped to the root of the output folder, 120 MB worth, at least some of which definitely shouldn't be there (more on that below).
Upon booting up the game while trying to run off of these extracted files, it starts checking files and tries to re-download missing ones (which of course fails since the game servers are long gone and it's set up to connect to localhost now anyway).
It hangs on a white screen while it does this, that's as far as it got before I eventually killed it.
BTW client/res/ui/ingame/hud.swf and client/res/ui/ingame/hud.gfx are interchangeable, and the game looks for one if the other can't be found (usually the game only contains the .gfx versions).
I found that hud.gfx and another related file, hud.xml, had been dumped to the root output folder. client/res/ui/ingame/hud.gfx as 000038ca.cfx, and client/res/ui/ingame/hud.xml as 000038de.xml.
I then checked to see if some of the missing animations (mf_a_g_360-throw.kf and mf_a_g_adventurer-shoot-loop-straightbow.kf) had ended up in the root of the output folder too, but couldn't find them (I sorted by size and there weren't even any other files in there with the same size).
ProfessorBrickkeeper then mentioned to me that he was looking in the extracted macros folder, and besides it missing quite a few files, he'd also found one (client/res/macros/sim_spaceshipcomplete.scm) that wasn't even a macro, but a DDS. I'd told quickbms to rename conflicting files instead of overwriting, so I took a look:
The first 1kb file is the correct macro file, the other three are completely unrelated textures, two of which are identical (side note: there are some duplicate textures in different locations in the game data normally, not sure about this one specifically though).
Similar things happened to many other files - for example, client/res/macros/primetime.scm is correct, but client/res/macros/primetime_00000001.scm is actually the file that should be located at client/res/scripts/client/ai/AG/L_AG_NPC_BECK.lua. But there's no file at that location, it's been confused with this macro instead.
Here's the pk files that contain primetime.scm and L_AG_NPC_BECK.lua, plus those two files themselves: https://www.dropbox.com/s/7xbmwq9bzzokn ... 2.zip?dl=0
I'm assuming that those missing animations I couldn't find have ended up confused with other files like L_AG_NPC_BECK.lua was.
I can try to figure out which pk contains hud.gfx and hud.xml later today if you need me to.
Upon booting up the game while trying to run off of these extracted files, it starts checking files and tries to re-download missing ones (which of course fails since the game servers are long gone and it's set up to connect to localhost now anyway).
Code: Select all
09:01:26 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_360-throw.kf'
09:01:26 3 RESOURCE Warning! File missing [fonts\ArialRoundedMTBold_A_14.nff]
09:01:26 3 RESOURCE Warning! File missing [fonts\ArialRoundedMTBold_A_35.nff]
09:01:26 3 RESOURCE Warning! File missing []
09:01:26 3 GUI File Failed to Open ui/ingame/hud.gfx
09:01:26 3 GUI Error: GFxLoader failed to open 'ui/ingame/hud.gfx'
09:01:26 3 RESOURCE Warning! File missing []
09:01:26 3 GUI File Failed to Open ui/ingame/hud.swf
09:01:26 3 GUI Error: GFxLoader failed to open 'ui/ingame/hud.swf'
09:01:26 3 GUI Unable to load scene HUD.
09:01:26 3 RESOURCE Warning! File missing []
09:01:26 3 GUI File Failed to Open ui/frontend/FEHelpScreen_I23.dds
09:01:26 3 GUI Error: Failed to load image 'ui/frontend/FEHelpScreen_I23.dds'
09:01:26 3 RESOURCE Warning! File missing []
09:01:26 3 GUI File Failed to Open ui/frontend/character_select_IB.dds
09:01:26 3 GUI Error: Failed to load image 'ui/frontend/character_select_IB.dds'
09:01:28 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:28 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_adventurer-shoot-loop-straightbow.kf'
09:01:30 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:30 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_blaster-master-dbl-jump.kf'
09:01:32 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:32 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_bow_charge.kf'
09:01:34 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:34 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_charge-slash-attack_04.kf'
09:01:36 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:36 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_handgun_ranged1-upperbody.kf'
09:01:38 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:38 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_pistol-charge-attack2.kf'
09:01:40 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:40 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_pocket-cannon-sequence.kf'
09:01:42 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:42 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_rifle-charge-attack.kf'
09:01:44 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:44 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_rifle-charge.kf'
09:01:46 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:46 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_rifle-ranged_01.kf'
09:01:48 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:48 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_skill-bomb-throw.kf'
09:01:50 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:50 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_slash-attack.kf'
09:01:52 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:52 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_slash-attack2.kf'
09:01:54 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:54 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_slash-attack3-upperbody.kf'
09:01:56 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:56 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_spear_attack_end-upperbody.kf'
09:01:58 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:01:58 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_spear_charge_attack-upperbody.kf'
09:02:00 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:02:00 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_spear_charge_attack.kf'
09:02:02 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:02:02 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_spinjitzu-spin-staff.kf'
09:02:04 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:02:04 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_twohand-attack-three-upperbody.kf'
09:02:06 5 SSLConnection - MakeRequest - Error (Can't perform HTTP request.) Error Code (12029)
09:02:06 3 RESOURCE HTTP error 0 when trying to download 'animations\minifig\attack\mf_a_g_twohand-attack-three.kf'It hangs on a white screen while it does this, that's as far as it got before I eventually killed it.
BTW client/res/ui/ingame/hud.swf and client/res/ui/ingame/hud.gfx are interchangeable, and the game looks for one if the other can't be found (usually the game only contains the .gfx versions).
I found that hud.gfx and another related file, hud.xml, had been dumped to the root output folder. client/res/ui/ingame/hud.gfx as 000038ca.cfx, and client/res/ui/ingame/hud.xml as 000038de.xml.
I then checked to see if some of the missing animations (mf_a_g_360-throw.kf and mf_a_g_adventurer-shoot-loop-straightbow.kf) had ended up in the root of the output folder too, but couldn't find them (I sorted by size and there weren't even any other files in there with the same size).
ProfessorBrickkeeper then mentioned to me that he was looking in the extracted macros folder, and besides it missing quite a few files, he'd also found one (client/res/macros/sim_spaceshipcomplete.scm) that wasn't even a macro, but a DDS. I'd told quickbms to rename conflicting files instead of overwriting, so I took a look:
The first 1kb file is the correct macro file, the other three are completely unrelated textures, two of which are identical (side note: there are some duplicate textures in different locations in the game data normally, not sure about this one specifically though).
Similar things happened to many other files - for example, client/res/macros/primetime.scm is correct, but client/res/macros/primetime_00000001.scm is actually the file that should be located at client/res/scripts/client/ai/AG/L_AG_NPC_BECK.lua. But there's no file at that location, it's been confused with this macro instead.
Here's the pk files that contain primetime.scm and L_AG_NPC_BECK.lua, plus those two files themselves: https://www.dropbox.com/s/7xbmwq9bzzokn ... 2.zip?dl=0
I'm assuming that those missing animations I couldn't find have ended up confused with other files like L_AG_NPC_BECK.lua was.
I can try to figure out which pk contains hud.gfx and hud.xml later today if you need me to.
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Interesting.
So I suppose that the 2 hashes are not the right things to look to retrieve the filenames or there is something I'm missing?
They are unique for each file so it seems all correct from the point of view of the parsing.
So I suppose that the 2 hashes are not the right things to look to retrieve the filenames or there is something I'm missing?
They are unique for each file so it seems all correct from the point of view of the parsing.
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
(Sorry for the slow reply, been focused on other things the past several days)
I'm certain the 2 hashes are definitely the right things to look for... Tonight I looked into two of the files that were being confused, primetime.scm and L_AG_NPC_BECK.lua (which, as said earlier, is being extracted as primetime_00000001.scm).
Here's primetime.scm's entry in trunk.txt:
I used Windows Grep to search through all of the .pk files in the game for the first two hashes, 8f0494da02ba0bd6ed8bfbb5e7f49fe7 and 113a6fa39e6a765bde21f30768bdd354 (the third hash doesn't seem to ever appear in the .PKs [I haven't seen it anywhere else actually], only the first two). They were found only once each, in macrox.pk.
Then I searched the PKs for the first two hashes of the misplaced lua file, 8ddd69151048099a8d4caa547d493c01 and 113edbf941fb9a21beb01a41fbb2ad06...
Each hash was only found once, in scripts.pk.
So if it's those two hashes being used to look up the name/path of each file, there doesn't seem to be any way to confuse files, aside from a bug in the script?... They're definitely unique for each file, and not found in multiple PKs either.
Also... What does the .pki file do, exactly? As far as I can tell, all that'd be needed for extraction is trunk.txt to get the file paths/names from, using the hashes in the PKs, unless there's more I don't know about.
Sorry I can't really be of more help with this... I'm not really experienced enough with programming or reverse engineering to do so. I appreciate you putting the work you have into this though.
Edit: Something I just noticed - whoops, there ARE files with the same first two hashes, if the files are identical but in different places. But in that case, the third hash is different. Example:
Edit again: Another example:
Edit agaaaain: Hm, just noticed that only the first entry for MegaMaelstrom2.dds was extracted, the one in client/res/mesh/auramar/creatures/. None of the others exist - in fact, the whole client/res/textures/auramar/creatures/ folder is missing, as MegaMaelstrom2.dds is the only file that's supposed to be in it, haha! And the first one listed has *six* duplicates extracted in that place... Similarly, only client/res/mesh/lup/env_nim_ag_proto_rock.dds exists, client/res/mesh/lup/showcase/env_nim_ag_proto_rock.dds doesn't. But there is one duplicate in the first location.
But still, the trouble with identical files in different locations doesn't explain how totally unrelated files like that macro and lua script are getting confused...
Edit AGAIN: Just remembered... The third hash is just an MD5 of the file's entry before that, aside from the preceding comma.
"5f04ee5bbe0c66fc121e06f85edfb4ff" is just what you get if you generate an MD5 of "client/res/animations/minifig/attack/mf_a_g_360-throw.kf,20876,9ef24738560fcb9fa9ba0e90364193e4,10984,3b17556f3cfdb497dc889899cb463ff6". I'd known this at one point when people first tried LU modding but forgot until now, whoops.
I'm certain the 2 hashes are definitely the right things to look for... Tonight I looked into two of the files that were being confused, primetime.scm and L_AG_NPC_BECK.lua (which, as said earlier, is being extracted as primetime_00000001.scm).
Here's primetime.scm's entry in trunk.txt:
Code: Select all
client/res/macros/primetime.scm,3847,8f0494da02ba0bd6ed8bfbb5e7f49fe7,683,113a6fa39e6a765bde21f30768bdd354,2502aa6c18808fbfc1bc3b91f768f82eI used Windows Grep to search through all of the .pk files in the game for the first two hashes, 8f0494da02ba0bd6ed8bfbb5e7f49fe7 and 113a6fa39e6a765bde21f30768bdd354 (the third hash doesn't seem to ever appear in the .PKs [I haven't seen it anywhere else actually], only the first two). They were found only once each, in macrox.pk.
Then I searched the PKs for the first two hashes of the misplaced lua file, 8ddd69151048099a8d4caa547d493c01 and 113edbf941fb9a21beb01a41fbb2ad06...
Code: Select all
client/res/scripts/client/ai/AG/L_AG_NPC_BECK.lua,431,8ddd69151048099a8d4caa547d493c01,264,113edbf941fb9a21beb01a41fbb2ad06,789a0d41c90eb8e37ccce7e89d8570feEach hash was only found once, in scripts.pk.
So if it's those two hashes being used to look up the name/path of each file, there doesn't seem to be any way to confuse files, aside from a bug in the script?... They're definitely unique for each file, and not found in multiple PKs either.
Also... What does the .pki file do, exactly? As far as I can tell, all that'd be needed for extraction is trunk.txt to get the file paths/names from, using the hashes in the PKs, unless there's more I don't know about.
Sorry I can't really be of more help with this... I'm not really experienced enough with programming or reverse engineering to do so. I appreciate you putting the work you have into this though.
Edit: Something I just noticed - whoops, there ARE files with the same first two hashes, if the files are identical but in different places. But in that case, the third hash is different. Example:
Code: Select all
client/res/mesh/lup/env_nim_ag_proto_rock.dds,524416,302ce039f25bf3bf582d2245dc8ba08a,278378,ccb4cbfaf2767e83dbd8be08c6506e4f,e9f95ceb7180f8f66def2f8594ce014b
client/res/mesh/lup/showcase/env_nim_ag_proto_rock.dds,524416,302ce039f25bf3bf582d2245dc8ba08a,278378,ccb4cbfaf2767e83dbd8be08c6506e4f,69360806e47ab50f1992e0e0c114a943Edit again: Another example:
Code: Select all
client/res/mesh/auramar/creatures/MegaMaelstrom2.dds,262272,be341eaeedce680be707e1bfc4b4b955,81762,977374fc0f5400ddebd6896151ef7304,47f1c709e48a5786451e3c794d73bc10
client/res/mesh/minifig/accessories/melee_wpn/MegaMaelstrom2.dds,262272,be341eaeedce680be707e1bfc4b4b955,81762,977374fc0f5400ddebd6896151ef7304,cef64c44d78a0ccffe08ce758aee8b97
client/res/mesh/ng_ninjago/ng_general/minifig/items/MegaMaelstrom2.dds,262272,be341eaeedce680be707e1bfc4b4b955,81762,977374fc0f5400ddebd6896151ef7304,c3448e7e6e47eb0023af6721cb167c00
client/res/textures/auramar/creatures/MegaMaelstrom2.dds,262272,be341eaeedce680be707e1bfc4b4b955,81762,977374fc0f5400ddebd6896151ef7304,9db3431925f59ec13c9648e4bc977a11Edit agaaaain: Hm, just noticed that only the first entry for MegaMaelstrom2.dds was extracted, the one in client/res/mesh/auramar/creatures/. None of the others exist - in fact, the whole client/res/textures/auramar/creatures/ folder is missing, as MegaMaelstrom2.dds is the only file that's supposed to be in it, haha! And the first one listed has *six* duplicates extracted in that place... Similarly, only client/res/mesh/lup/env_nim_ag_proto_rock.dds exists, client/res/mesh/lup/showcase/env_nim_ag_proto_rock.dds doesn't. But there is one duplicate in the first location.
But still, the trouble with identical files in different locations doesn't explain how totally unrelated files like that macro and lua script are getting confused...
Edit AGAIN: Just remembered... The third hash is just an MD5 of the file's entry before that, aside from the preceding comma.
Code: Select all
client/res/animations/minifig/attack/mf_a_g_360-throw.kf,20876,9ef24738560fcb9fa9ba0e90364193e4,10984,3b17556f3cfdb497dc889899cb463ff6,5f04ee5bbe0c66fc121e06f85edfb4ff"5f04ee5bbe0c66fc121e06f85edfb4ff" is just what you get if you generate an MD5 of "client/res/animations/minifig/attack/mf_a_g_360-throw.kf,20876,9ef24738560fcb9fa9ba0e90364193e4,10984,3b17556f3cfdb497dc889899cb463ff6". I'd known this at one point when people first tried LU modding but forgot until now, whoops.
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
*edit*: the two hashes are calculated on the uncompressed and compressed data.
The third one is probably the name hash.
The following comes from anim1.pk:
As visible only DUMMY1 and DUMMY5 differ but they don't represent something that can be used in trunk.txt (DUMMY1 can be a 32bit CRC but in trunk.txt you have an hash).
So I opted for the simplest and correct solution: removing the elements from the array when I use them so that now there will be no longer files with the same name.
Script 0.3
The third one is probably the name hash.
The following comes from anim1.pk:
Code: Select all
. 001252e6 get DUMMY1 0x0f12146b 4
. 001252ea get DUMMY2 0xffffffff 4
. 001252ee get DUMMY3 0xffffffff 4
. 001252f2 get SIZE 0x0000055e 4
. 001252f6 getdstr HASH1 "5542a313ada7b6862888de2ebe8293ee" 32
35 35 34 32 61 33 31 33 61 64 61 37 62 36 38 36 5542a313ada7b686
32 38 38 38 64 65 32 65 62 65 38 32 39 33 65 65 2888de2ebe8293ee
. 00125316 get DUMMY5 0x047f9400 4
. 0012531a get ZSIZE 0x00000161 4
. 0012531e getdstr HASH2 "87e65bb420d49b3320adfc7f58620814" 32
38 37 65 36 35 62 62 34 32 30 64 34 39 62 33 33 87e65bb420d49b33
32 30 61 64 66 63 37 66 35 38 36 32 30 38 31 34 20adfc7f58620814
. 0012533e get DUMMY7 0x00e1ce00 4
. 00125342 get OFFSET 0x000204fc 4
. 00125346 get DUMMY9 0x0309a800 4
000204fc 1374 client/res/animations/character-create/hair/cc_hair_01.settings
. 001253ae get DUMMY1 0x140c32d8 4
. 001253b2 get DUMMY2 0xffffffff 4
. 001253b6 get DUMMY3 0xffffffff 4
. 001253ba get SIZE 0x0000055e 4
. 001253be getdstr HASH1 "5542a313ada7b6862888de2ebe8293ee" 32
35 35 34 32 61 33 31 33 61 64 61 37 62 36 38 36 5542a313ada7b686
32 38 38 38 64 65 32 65 62 65 38 32 39 33 65 65 2888de2ebe8293ee
. 001253de get DUMMY5 0x047f9300 4
. 001253e2 get ZSIZE 0x00000161 4
. 001253e6 getdstr HASH2 "87e65bb420d49b3320adfc7f58620814" 32
38 37 65 36 35 62 62 34 32 30 64 34 39 62 33 33 87e65bb420d49b33
32 30 61 64 66 63 37 66 35 38 36 32 30 38 31 34 20adfc7f58620814
. 00125406 get DUMMY7 0x00e1ce00 4
. 0012540a get OFFSET 0x000288e0 4
. 0012540e get DUMMY9 0x0309a800 4
000288e0 1374 client/res/animations/character-create/hair/cc_hair_01.settingsAs visible only DUMMY1 and DUMMY5 differ but they don't represent something that can be used in trunk.txt (DUMMY1 can be a 32bit CRC but in trunk.txt you have an hash).
So I opted for the simplest and correct solution: removing the elements from the array when I use them so that now there will be no longer files with the same name.
Script 0.3
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Alright... It seems to be getting some things more correct now, like L_AG_NPC_BECK.lua is now in its correct place. There's still a lot of files ending up in the places of other files though... Random example, client/res/textures/march2011content/ui/news/loading_spinjitzu.dds is being extracted as client\res\scripts\ai\AG\L_AG_DARKLING_MECH_FOR_SHIP.lua. There's also now 6,311(!) files at the root of the output folder (including hud.gfx and hud.xml as before).
At this point I really don't know what's going on to possibly cause this, haha. It seems to be working correctly on *most* files...
At this point I really don't know what's going on to possibly cause this, haha. It seems to be working correctly on *most* files...
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Script 0.3.1, now I think we have the 100% of files
-
Terrev
- Posts: 8
- Joined: Mon Jul 25, 2016 9:27 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
It's looking better, I'm seeing files extract that weren't there before (some I didn't know were even *in* the PKs, haha)!... Still not 100% correct though. But, this happened:
Part way through extracting I got that. Didn't look right, so I looked in trunk.txt...
... It seems to be getting confused on paths/files with spaces? Which are relatively few, but they do exist.
I stopped the extraction there and looked at what it had extracted so far. There's still files getting misplaced - I opened client\res\animations\minifig\attack\mf_a_g_shurikun-attack-two.kf after noticing its huge file size, and got a model for a spaceship interior, lol!
Again, I do appreciate all you're doing with this. Sorry it's been such a hassle!
Part way through extracting I got that. Didn't look right, so I looked in trunk.txt...
Code: Select all
client/res/forkp/effects/FVracetrack/environment/racing_fv_blowing candle/racing_fv_blowing candle.psb,1184,2b407e5d6a3aa479ed3ac46fc202d9d0,425,65f127c36eac48ba0bf8e03a1da92b3f,1f9c4d53408befd1ce32c0c61861d20c
client/res/forkp/effects/FVracetrack/environment/racing_fv_blowing candle/racing_fv_blowing candle.txt,311,a86ef9ce57ac3ee64e2e7d78f97b4846,167,4ac5e5afb764ed94c5784160577bd974,1f78d54f9f2c9cf2abffa0e274b178bf... It seems to be getting confused on paths/files with spaces? Which are relatively few, but they do exist.
I stopped the extraction there and looked at what it had extracted so far. There's still files getting misplaced - I opened client\res\animations\minifig\attack\mf_a_g_shurikun-attack-two.kf after noticing its huge file size, and got a model for a spaceship interior, lol!
Again, I do appreciate all you're doing with this. Sorry it's been such a hassle!
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
The problem with that "racing fv" file is in the spaces inside the name, I will check how I can improve the script to consider only the commas.
Regarding that other file I have no idea.
Regarding that other file I have no idea.
-
aluigi
- Site Admin
- Posts: 12984
- Joined: Wed Jul 30, 2014 9:32 pm
Re: LEGO Universe - .pk & sd0 (UPDATED)
Script 0.3.2, I simply replaced the spaces with underscore, easy solution.