PhyreEngine 2D textures (.phyre) (Noesis: 0.6.2/QBMS: 0.5.1 - abandoned)

[TheUkrainianBard]

NOTICE: Consider these scripts abandoned.

Due to my recent frustrations with extracting textures from PhyreEngine-based games that use stock PhyreEngine asset format I've decided to make a Noesis script that parses the structure (at least important parts of it) and allows for previewing/extracting 2D textures.

Originally based on Acewell's script for Trails of Cold Steel II.

What the script allows to do: preview and export most supported 2D textures from PC/PS3/PSVita/PS4/Switch in stock PhyreEngine asset format ("RYHP"/"PHYR" magic).
What the script doesn't allow to do: preview and export cubemaps (PTextureCubeMap), 3D textures (PTexture3D), atlas maps, models, shaders, fonts etc.

Titles confirmed working:
PC titles:

• Secret Ponchos
• The Legend of Heroes: Trails of Cold Steel I & II: -1-; -2-
• Tokyo Xanadu eX+
• Final Fantasy X HD
• Final Fantasy XII: The Zodiac Age
• SAO: Re: Hollow Fragment
• SAO: Hollow Realization
• Secret of Mana Remake

PS3 titles:

• OreImo Happy enD: -1-; -2-
• Super Heroine Chronicle
• Hajime no Ippo: The Fighting!
• Dragon Quest Builders
• The Legend of Heroes: Trails of Cold Steel II

PSVita titles:

• Labyrinth of Refrain: Coven of Dusk - one sample

PS4 titles:

• Final Fantasy XII: The Zodiac Age
• Tokyo Xanadu eX+: -1-; -2-
• Final Fantasy X HD
• 英雄伝説 閃の軌跡III (Sen no Kiseki III aka Trails of Cold Steel III)
• 限界凸城 キャッスルパンツァーズ (Genkai Tokki: Castle Panzers)

Switch titles:

• Final Fantasy X HD: -1-; -2-; -3-

[TheUkrainianBard]

Changelog for Noesis script:

Code: Select all

0.6.2 - GNM: Now using pitch defined in GNF parameters for some cases
        inc_ps4untiling: added pitch parameter
0.6.1 - NVN: Removed hardcoded block_height
           | Now it's computed
           | Unswizzling might still break under specific conditions:
           | Height is non-power-of-2 and less than 128 pixels/blocks
0.6 - NVN: Added Nintendo Switch support
         | Added Switch unswizzling support
         | Can be used separately
         | from inc_switchunswizzling import *
         | imageUnswizzleSwitch(input_array, width, height, bytes_per_pixel, block_height)
         | adapted from an old revision of Yuzu, a Nintendo Switch Emulator
         | https://github.com/yuzu-emu/yuzu
         | might break
      inc_ps4untiling: renamed "imageFromPeanoOrder" function to (a more logical) "imageUntilePS4"
                     | imageFromPeanoOrder is still supported
                     | now it's just a wrap for imageUntilePS4
0.5.3 - ALL: Other code cleanup
0.5.2 - ALL: No more hardcoded values
           | Rewrite of reader process
           | Mipmaps are now removed on passthrough
           | Unified texture processing as much as possible
           | Other code cleanup (a lot of it)
0.5.1 - ALL: Fixed filename export
           | Code cleanup (a lot of it)
0.5 - GNM: Bypassed broken image unswizzling
         | Noesis' GNF image processing is broken with non-power-of-2 width images
         | Can be used separately
         | from inc_ps4untiling import *
         | imageFromPeanoOrder(input_array, width, height, bytes_per_pixel)
         | Adapted from Scarlet and fixed a bug
         | https://github.com/xdanieldzd/Scarlet/
      ALL: Multiple concatenated files support
         | PFsBin support (by skipping it)
0.4.3 - GCM, DX11: Tremendously sped up byte operations used for:
                 | 1) Upconverting L8->RGB8 and LA88->RGBA8
                 | By using slice assignment: https://stackoverflow.com/a/36096337
                 | 2) Byteswapping ARGB1555/ARGB4444
        DX11: Added RGBA16F support
        ALL: Code cleanup (by using Python lists - still WIP)
0.4.2 - GCM: Fixed ARGB1555 and ARGB4444 decoding
           | Fixed RGBA8/ARGB8 decoding (0.4 regression fix)
           | Added L8 and LA88 support
        GCM, DX11: L8 and LA88 are upconverted to RGB and RGBA respectively so now they are viewed/exported as proper grayscale
                 | Workaround for no grayscale support in imageDecodeRaw
        ALL: Eliminated almost all hardcoding
           | Added "KeepImportName" option
           | Will keep the right import filename for single texture files
           | Might not keep the right import filename for multiple texture files
           | Changed "KeepBlockCompression" option to "KeepDXT"
           | Noesis limitation, cannot preview/export as BC5/7 in any case
           | Forces decoding to RGBA
0.4.1 - GCM: Re-added DXT3 support 
           | 0.4 regression fix
        GCM, DX11: Added experimental "KeepBlockCompression" option
                 | May not work for more advanced BCns, like BC5/7
        ALL: Made parsing a bit less hardcoded; code cleanup
0.4 - GCM: Added multiple textures in single file support
         | Added ARGB1555 and ARGB4444 support
      GCM, GNM: Made parsing a bit less hardcoded
0.3 - GXM: Added support via importing a complete GXT image present in asset structure
0.2.3 - ALL: Added some comments; code cleanup
0.2.2 - ALL: Added additional safeguards against models and shaders; code cleanup
0.2.1 - GNM: Fixed *everything* (hopefully) by using GNF image header data inside Phyre structure
0.2 - GNM: Added support for DXT1/3/5/ARGB8 textures via faking a GNF image header and importing as a GNF image
0.1.5 - ALL: Dropped ".dds."/".png."/".bmp." file name requirement
           | Now it's just ".phyre" extension
0.1.4 - DX11: Added BC7 support
0.1.3 - GNM: Added Phyre structure read support
           | Cannot show them properly yet
0.1.2 - DX11: Added BC5 and A8 support
        ALL: Added a safeguard against cubemaps
0.1.1 - Code cleanup
0.1 - Initial release

Changelog for QuickBMS script:

Code: Select all

0.5.1 - ALL: Fixed filename export
0.5 - GCM: Fixed ARGB1555 and ARGB4444 decoding
         | last 128 bytes or 64 pixels were not byteswapped resulting in garbage
         | Added output modes - DDS or RAW
      GNM: Added GNF image untiling
         | VERY SLOW
         | ~2.5 pixels/blocks per MHz per second on Core i5 3rd gen
         | Adapted from Scarlet and fixed a bug
         | https://github.com/xdanieldzd/Scarlet/
         | Added output modes - DDS, RAW or GNF
         | Added configurable data offset for GNF
      ALL: Massive rewrite of output functions
         | Multiple concatenated files support
         | Added optional DX10 DDS output
         | Automatically forced/reverted when needed (BC7/LA88 respectively for ex.)
         | Added ARGB8_SRGB and RGBA16F support
         | PFsBin support (by skipping it)
0.4.2 - Initial release
        | Functionally on par with 0.4.2 for Noesis

[akderebur]

SAO has lots of textures in BC7 format, so you might want to add support for that. I don't know if noesis can decode bc7 though.

Also if your script works with SAO, it should also work with the Secret of Mana remake on pc. You can add that to the list.

[TheUkrainianBard]

Thank you.
Noesis can decode BC7:

Code: Select all

output = rapi.imageDecodeDXT(input, Width, Height, noesis.FOURCC_BC7)

Could you confirm it working with those SAO:HR BC7 textures and Secret of Mana remake?

The list is non-exhaustive but what I've personally confirmed working. Ideally this script should work with all PhyreEngine texture assets (at least on PC and PS3 for now).

[akderebur]

Too lazy to test it on files now, moved some of them to external hdd. I worked on both sao and secret of mana files and their structures are exactly the same. So it will work

[TheUkrainianBard]

I wonder if this is some sorta tiling taking place on PS4...
UPD: Well, I have extracted something somehow and those are GNF images

UPD2 (2019/04/14): removed attached images.

[klose_rinz]

Thank you! Already solved my problem~

[TheUkrainianBard]

@klose_rinz
Hm, is that from PS4 re-release of Sen no Kiseki aka Sen no Kiseki Kai ?

[klose_rinz]

@klose_rinz
Hm, is that from PS4 re-release of Sen no Kiseki aka Sen no Kiseki Kai ?

It's from Tokyo Xanadu ex+. Sen no kiseki 3 and Sen no kiseki Kai are on 4.70+ version, and we cannot get the files.

Your code works well with the .dds.phyre file from this game.
But there are some problems when opening .png.phyre(ARGB8) files.

I change the code with the following, then I can get the picture, but the colors are not right.
You can view the vis_127 in the attachment. The left above of the picture is dark sky with nearly pure black color. But in the extracted picture, it's two different bandings with green and blue.
So I think the order of the color is not the problem, maybe there are some tricks in the pixels which are different from PS3 swizzling.

Code: Select all

    if Platform == "GNM\x02":
        FakeGNF = FakeGNFHeader + data
        FakeGNF = rapi.loadTexByHandler(FakeGNF, ".gnf")
        FakeGNF.name = "test.gnf"
        data = rapi.imageGetTexRGBA(FakeGNF)
        try:
            if imgFmt == "ARGB8":
               if IsSwizzled == 1:
                  # data = rapi.imageFromMortonOrder(data, imgWidth, imgHeight, 4)
               data = rapi.imageDecodeRaw(data, imgWidth, imgHeight, "b8 g8 r8 a8")
            data = rapi.imageFlipRGBA32(data, imgWidth, imgHeight, 0, 1)
            texList.append(NoeTexture(rapi.getInputName(), imgWidth, imgHeight, data, noesis.NOESISTEX_RGBA32))
        except:
            texList.append(FakeGNF)
    else:
        data = rapi.imageFlipRGBA32(data, imgWidth, imgHeight, 0, 1)
        texList.append(NoeTexture(rapi.getInputName(), imgWidth, imgHeight, data, noesis.NOESISTEX_RGBA32))

    return 1

Also, if I manually change IsSwizzled to 1, the code "data = rapi.imageFromMortonOrder(data, imgWidth, imgHeight, 4)" will get exception.

[TheUkrainianBard]

Yeah, that's the problem with having a very limited amount of samples. I didn't realise that phyre structure had all necessary GNF image data (I kinda did realise for GCM, but there was no real need to use them)...
Attached file is what I got after I've actually used that GNF image data.

I've updated the script in the first post.

UPD2 (2019/04/14): removed attached image.

[klose_rinz]

Yeah, that's the problem with having a very limited amount of samples. I didn't realise that phyre structure had all necessary GNF image data (I kinda did realise for GCM, but there was no real need to use them)...
Attached file is what I got after I've actually used that GNF image data.

I've updated the script in the first post.

Many thanks~

[wren44]

i would like to say thanks for this plugin, i used it for Secret of Mana and Tokyo Xanadu, very pleased with the outcome,
i noticed you added support for GXM for the PSVita, so i tried to use it on Dragon Quest Builders as the file name says it is a dds.GXM.phyre, but i get this error

Output extension has set output file type to:
.dds - DDS Image
Detected file type: PhyreEngine texture
[i] Platform: 'GXM\x01'.
Traceback (most recent call last):
File "C:\noesisv4176\plugins\python\tex_PhyreEngine_phyre.py", line 141, in noepyLoadRGBA
print("[i] Phyre asset import name: "+m_buffer)
TypeError: must be str without null bytes, not str

i get this kind of error on some files i try, i also tried the ps3 version which is dds.GCM.phyre, but that also puts out this error

Output extension has set output file type to:
.dds - DDS Image
Detected file type: PhyreEngine texture
[i] Platform: 'GCM'.
Traceback (most recent call last):
File "C:\noesisv4176\plugins\python\tex_PhyreEngine_phyre.py", line 141, in noepyLoadRGBA
print("[i] Phyre asset import name: "+m_buffer)
TypeError: must be str without null bytes, not str


but some images view and can be exported, just not the BG folder stuff which i am guessing that it is some kind of archives? if so how do i go about getting the images inside it out if it is possible, thank you advance.

[TheUkrainianBard]

Please post files that don't work in any kind of archive (zip for ex). Or try commenting the line 141 by putting # at the very start of it.

[wren44]

thank you, it allowed me to view some textures, misaligned though, easy fix, though it won't veiw the other textures in these files, only some, i did find out that there is more than one format in there, using texture-finder, and was able to get more that way, allot more work but still could find more, here is a sample of a small file that i cannot view, and couldn't view with texture-finder either, maybe not a texture not sure, i know models are stored in there too

3mb file for blocks
1mb for environments

i am hoping to get more textures out, just would like an easier way, i have seen a few programs for other phyre engine games that gets just about everything out, just nothing for Dragon Quest Builders, only to unpack the psarc file, which is where i am now, and i know it uses phyre, the ps3 version is GCM and the vita version is GXM, but i am only able to get the ps3 version so far, some of the vita but not much, i don't mind which ever version works as long as i am happy to get get the files out, anyways thank you for the help it has gotten me more files, ill leave the samples here in hopes more people maybe can get them out...

[TheUkrainianBard]

Here's a temporary solution specifically for those "one phyre file - multiple 2D textures inside" like your "blocks" file. But there is an enormous regression - it can't open anything else from PS3, so you have to shuffle files from this and the first post.
After I'll fix the regression - I'll delete the file from this post.

Even more samples are appreciated - maybe like a complete folder archived.

[wren44]

omg thank you so much, it even got the bump map image too, i wasn't expecting to get that, but now very happy to have,

here is the whole data file, everything is in the buildgra folder for images that i am aware of

DQB 226mb archive

its large but it is everything, anyways thank you so much for helping, even though it is a 2 year old game.

[Rainx]

TheUkrainianBard, can you add support for this sample:

[TheUkrainianBard]

@Rainx

[Rainx]

@Rainx

Error:

Code: Select all

WARNING: Could not load external texture
Traceback (most recent call last):
  File "D:\noesis\plugins\python\tex_PhyreEngine_phyre.py", line 276, in noepyLoadRGBA
    FakeGNF.name = "test.gnf"
AttributeError: 'NoneType' object has no attribute 'name'

[TheUkrainianBard]

Strange, it worked for me and there was no regression for my PS4 samples.

Update: I've fixed regressions, first post updated.

UPD2 (2019/04/14): removed attachments.