[REDSHIFT]FBX Import Helper

[Atom]

I created a short python script to help out with the excessive nodes that are presented when importing detailed FBX files. My example is a vehicle with 278 object nodes referencing 37 materials inside the materials subnet generated by the import.

This script scans the materials subnet and creates a new object level geo for each material it finds. Inside this new geo node it creates an ObjectMerge node and populates the node with all object references to the FBX material. It assigns the new material to this new geo node that points to /shop instead of the FBX materials subnet. Then it reviews the FBX materials and creates a new Redshift material for each FBX material detected. It scans the FBX Surface node and extracts a few parameters like diffuse color, specular etc...

The net result is that I only have 37 nodes to manage instead of 278 after running the script. Also my nodes have Redshift placeholder materials assigned so I can get right to rendering.

 

Add this code to a new shelf button and adjust the paths, at the bottom of the script, to point to your FBX subnet. The texture path is not really used at this time.

# Scan a FBX subnet for materials.
# Create a geo object with an object merge for each object that references the material.
# Create a place holder Redshift material by reviewing the FBX materials in the subnet.
# Atom  08-22-2018
#	10-14-2018

import hou, os, re

def returnValidHoudiniNodeName(passedItem):
    # Thanks to Graham on OdForce for this function!
    # Replace any illegal characters for node names here.
    return re.sub("[^0-9a-zA-Z\.]+", "_", passedItem)
    
def createRedshiftImageMapMaterial(passedSHOP, passedImageFilePath, passedName, passedDiffuse=[0,0,0], passedSpecular=[0,0,0], passedWeight=0.1, passedRoughness=0.23, passedIOR=1.0, passedOpacity=1.0):
    #print "->%s [%s] [%s]" % (passedSHOP, passedImageFilePath, passedName)
    rs_vop = hou.node(passedSHOP).createNode("redshift_vopnet",passedName)
    if rs_vop != None:
        rs_output = hou.node("%s/%s/redshift_material1" % (passedSHOP, passedName))  # Detect the default closure node that should be created by the redshift_vopnet.
        if rs_output != None:
            # Create.
            rs_mat = rs_vop.createNode("redshift::Material","rs_Mat")
            if rs_mat != None:
                # Set passed values.
                rs_mat.parm("diffuse_colorr").set(passedDiffuse[0])
                rs_mat.parm("diffuse_colorg").set(passedDiffuse[1])
                rs_mat.parm("diffuse_colorb").set(passedDiffuse[2])
                rs_mat.parm("refl_colorr").set(passedSpecular[0])
                rs_mat.parm("refl_colorg").set(passedSpecular[1])
                rs_mat.parm("refl_colorb").set(passedSpecular[2])
                rs_mat.parm("refl_weight").set(passedWeight)
                rs_mat.parm("refl_roughness").set(passedRoughness) 
                if passedIOR ==0:
                    # A zero based IOR means activate mirror mode for the reflection section.
                    rs_mat.parm("refl_fresnel_mode").set("1") 
                    rs_mat.parm("refl_brdf").set("1") 
                    rs_mat.parm("refl_reflectivityr").set(0.961998) 
                    rs_mat.parm("refl_reflectivityg").set(0.949468)
                    rs_mat.parm("refl_reflectivityb").set(0.91724)
                    rs_mat.parm("refl_edge_tintr").set(0.998643) 
                    rs_mat.parm("refl_edge_tintg").set(0.998454)
                    rs_mat.parm("refl_edge_tintb").set(0.998008)
                    rs_mat.parm("refl_samples").set(128) 
                    rs_mat.parm("diffuse_weight").set(0)
                else:
                    rs_mat.parm("refl_ior").set(passedIOR)
                rs_mat.parm("opacity_colorr").set(passedOpacity)
                rs_mat.parm("opacity_colorg").set(passedOpacity)
                rs_mat.parm("opacity_colorb").set(passedOpacity)
                
                rs_tex = rs_vop.createNode("redshift::TextureSampler",returnValidHoudiniNodeName("rs_Tex_%s" % passedName))
                if rs_tex != None:
                    # Wire
                    try:
                        rs_output.setInput(0,rs_mat)
                        can_continue = True
                    except:
                        can_continue = False
                    if can_continue:
                        if passedImageFilePath.find("NOT_DETECTED")==-1:
                            # Only plug in texture if the texture map was specified.
                            rs_mat.setInput(0,rs_tex)                       # input #0 is diffuse color.
                        extension = os.path.splitext(passedImageFilePath)[1]
                        files_with_alphas = [".png",".PNG",".tga",".TGA",".tif",".TIF",".tiff",".TIFF",".exr",".EXR"]
                        if extension in files_with_alphas:
                            # Place a sprite after the rsMaterial to implment opacity support.
                            rs_sprite = rs_vop.createNode("redshift::Sprite",returnValidHoudiniNodeName("rs_Sprite_%s" % passedName))
                            if rs_sprite != None:
                                rs_sprite.parm("tex0").set(passedImageFilePath)    # set the filename to the texture.
                                rs_sprite.parm("mode").set("1")
                                rs_sprite.setInput(0,rs_mat)
                                rs_output.setInput(0,rs_sprite)
                                #rs_mat.setInput(46,rs_tex)                  # input #46 is opacity color (i.e. alpha).

                        rs_tex.parm("tex0").set(passedImageFilePath)    # set the filename to the texture.
                        
                        # Remove luminosity from texture using a color corrector.
                        rs_cc = rs_vop.createNode("redshift::RSColorCorrection",returnValidHoudiniNodeName("rs_CC_%s" % passedName))
                        if rs_cc != None:
                            rs_cc.setInput(0,rs_tex)
                            
                            rs_cc.parm("saturation").set(0)
                            # Add a slight bump using the greyscale value of the diffuse texture.
                            rs_bump = rs_vop.createNode("redshift::BumpMap",returnValidHoudiniNodeName("rs_Bump_%s" % passedName))
                            if rs_bump != None:
                                rs_bump.setInput(0,rs_cc)
                                rs_bump.parm("scale").set(0.25)          # Hard coded, feel free to adjust.
                                rs_output.setInput(2,rs_bump)
                                                
                        # Layout.
                        rs_vop.moveToGoodPosition() 
                        rs_tex.moveToGoodPosition()
                        rs_cc.moveToGoodPosition() 
                        rs_bump.moveToGoodPosition()
                        rs_mat.moveToGoodPosition()
                        rs_output.moveToGoodPosition()
                else:
                    print "problem creating redshift::TextureSampler node."
            else:
                print "problem creating redshift::Material node."
        else:
            print "problem detecting redshift_material1 automatic closure."
    else:
        print "problem creating redshift vop net?"

def childrenOfNode(node, filter):
    # Return nodes of type matching the filter (i.e. geo etc...).
    result = []
    if node != None:
        for n in node.children():
            t = str(n.type())
            if t != None:
                for filter_item in filter:
                    if (t.find(filter_item) != -1):
                        # Filter nodes based upon passed list of strings.
                        result.append((n.name(), t))
                    result += childrenOfNode(n, filter)
    return result
    
def groupByFBXMaterials(node_path, rewrite_original=False):
    lst_geo_objs = []
    lst_fbx_mats = []
    s = ""
    
    material_nodes = childrenOfNode(hou.node("%s/materials" % node_path),["Shop material"]) #Other valid filters are Sop, Object, cam.
    for (name, type) in material_nodes:
        node_candidate = "%s/%s" % ("%s/materials" % node_path, name)
        n = hou.node(node_candidate)
        if n !=None:
            lst_fbx_mats.append(node_candidate)

    object_nodes = childrenOfNode(hou.node(node_path),["Object geo"]) #Other valid filters are Sop, Object, cam.
    for (name, type) in object_nodes:
        node_candidate = "%s/%s" % (node_path, name)
        n = hou.node(node_candidate)
        if n !=None:
            lst_geo_objs.append(node_candidate)

    # Make an object geo node for each material detected.
    # Inside the object will reside an object merge to fetch in each object that references the material.
    root = hou.node("/obj")
    if root != None:
        for mat in lst_fbx_mats:
            mat_name = os.path.basename(mat)
            shader_name = "rs_%s" % mat_name
            geo_name = "geo_%s" % mat_name
            
            '''
            node_geo = root.createNode("geo", geo_name)
            if node_geo:
                # Delete the default File node that is automatically created as well.
                if (len(node_geo.children())) > 0:
                    n = node_geo.children()[0]
                    if n:
                        n.destroy()
                node_geo.parm("shop_materialpath").set("/shop/%s" % shader_name)
                node_obm = node_geo.createNode("object_merge","object_merge1")
                if node_obm != None:
                    p = node_obm.parm("objpath1")
                    all_obj = ""
                    for obj in lst_geo_objs:
                        temp_node = hou.node(obj)
                        if temp_node != None:
                            smp = temp_node.parm("shop_materialpath").eval()
                            if smp.find(mat_name) != -1:
                                all_obj += "%s " % obj
                    p.set(all_obj)
                    node_obm.parm("xformtype").set(1)
            '''        
            # Make a place holder Redshift material by reviewing the FBX material.
            opacity = 1.0
            ior = 1.025
            reflection_weight = 0.1
            reflection_roughness = 0.23
            diffuse_color = [0,0,0]
            specular_color = [0,0,0]
            
            
            # Typically the FBX Surface Shader is the second node created in the FBX materials subnet.
            n = hou.node(mat).children()[1]
            if n != None:
                r = n.parm("Cdr").eval()
                g = n.parm("Cdg").eval()
                b = n.parm("Cdb").eval()
                diffuse_color = [r,g,b]
                sm = n.parm("specular_mult").eval()
                if sm > 1.0: sm = 1.0
                reflection_weight = 1.0-sm
                if (sm==0) and (n.parm("Car").eval()+n.parm("Cdr").eval()==2):
                    # Mirrors should use another Fresnel type.
                    ior=0
                r = n.parm("Csr").eval()
                g = n.parm("Csg").eval()
                b = n.parm("Csb").eval()
                specular_color = [r,g,b]
                opacity = n.parm("opacity_mult").eval()
                reflection_roughness = n.parm("shininess").eval()*0.01  
                em = n.parm("emission_mult").eval()
                if em > 0:
                    # We should create an rsIncandescent shader, using this color, instead.
                    r = n.parm("Cer").eval()
                    g = n.parm("Ceg").eval()
                    b = n.parm("Ceb").eval()
                    
                # Try to fetch the diffuse image map, if any.
                tex_map = n.parm("map1").rawValue()
                if len(tex_map) > 0:
                    pass
                else:
                    tex_map = "%s/%s" % (texture_path,"NOT_DETECTED")
                
                createRedshiftImageMapMaterial("/shop", tex_map, shader_name, diffuse_color, specular_color, reflection_weight, reflection_roughness, ior, opacity)
   
        if rewrite_original:
            # Re-write the original object node's material reference to point to the Redshift material.
            for obj in lst_geo_objs:
                node_geo = hou.node(obj)
                if node_geo:
                    m = node_geo.parm("shop_materialpath").eval()
                    if len(m):
                        mat_name = os.path.basename(m)
                        shader_name = "/shop/rs_%s" % mat_name
                        # To do this right, we need to add a material node to the end of the network and populate it with the shop_materialpath value.
                        node_display = node_geo.displayNode()
                        if node_display != None:
                            node_mat = node_geo.createNode("material","material1")  # Create new node.
                            if node_mat != None:
                                node_mat.parm("shop_materialpath1").set(shader_name)
                                node_mat.setInput(0,node_display)                   # Wire it into the network.
                                node_mat.setDisplayFlag(True)                       # Move the display flag to the new node.
                                node_mat.setRenderFlag(True)                        # Move the render flag to the new node.
                                node_mat.moveToGoodPosition()
# Program starts here.
texture_path = '/media/banedesh/Storage/Documents/Models/Ford/Ford_F-150_Raptor_2017_crewcab_fbx'   #Not really used yet.
fbx_subnet_path = "/obj/Container_Ship_Generic_FBX"
groupByFBXMaterials(fbx_subnet_path, True)

 

 

Edited October 14, 2018 by Atom

[Atom]

I added a new feature to the script, which can be disabled or enabled by passing a True/False flag. This new feature allows you to re-write the original object nodes, inside the FBX subnet, with the new Redshift materials. So if you want to work with the original nodes, but assign them all new materials, try this feature.

Here is the result from running the script. The FBX materials are translated into Redshift equivalents. The script tries to guess mirror types and assigns an alternate fresnel mode in that case. For the headlights, I did manually choose the Glass preset inside the generated rsMaterial.

Edited August 24, 2018 by Atom

[nbyte]

Hey Atom,

I like your python script...

I tried to implement to read the fbx_subnet_path from selection but with no luck.

Here is what I tried:

sel = hou.selectedNodes()
Mypath = sel[0].path
texture_path = '/media/banedesh/Storage/Documents/Models/Ford/Ford_F-150_Raptor_2017_crewcab_fbx'   #Not really used yet.
fbx_subnet_path = str(Mypath)
groupByFBXMaterials(fbx_subnet_path, True)

[Atom]

Try print MyPath to see what you are passing? Maybe you have a trailing slash or something...

This code will only work with a single fbx subnet supplied.

Edited August 23, 2018 by Atom

[nbyte]

I tried that and it seems the path is correct.

If I uses the printed path in quotes - like you did in your script - everythings works!

Here is what I get when I print Mypath:

<bound method ObjNode.path of <hou.ObjNode of type subnet at /obj/SlopeHouse_FBX>>
 

[Atom]

Oh yeah, Houdini treats all parameters as functions so try...

Mypath = sel[0].path()

 

[nbyte]

Great it works ....

What is the difference between  .path() and .path?

[Atom]

.path fetches the code based object while .path() fetches the result of invoking the code based object. My guess is it basically runs the __INIT__.

[nbyte]

Thank you for the explanation Atom.

[PJM33]

Hey Atom!

Thanks so much for this script. It's been a help and step forward to the community.

I tried it but i'm not sure if I'm doing it correctly. I'm using Houdini. I imported a cg model that I downloaded from cg trader in fbx format. Houdini and mantra render the materials on the model perfectly but redshift is making me jump through hoops. 

I created a new shelf tool with your script and it moved the geo nodes within the fbx file in the obj level but it only moved the geo at the top of the ship and the geo nodes are missing their materials. When I try to render it either crashes or there is a render error and it doesn't render. 

I contacted Redshift and they said that 'Redshift only can work with its own VOP shader nodes, so you are going to need to convert the materials. Of course, you can use the texture maps provided with the models, but using the Redshift texture sampler VOP node to apply them.' 

Any ideas what I can do to get redshift to read and render my cg model's geo and materials?

 

[Atom]

With the new re-write material feature, you can throw away all those top level OBJ files unless you have an additional use for them.

Dive inside the FBX subnet and then inside one of the geometry objects. Did the script add a new material node to the end of the network? If so, is the material name on that node pointing to a Redshift material?

The script does not detect subnets inside subnets. It assumes all objects in the FBX reside at the top level of the master FBX subnet.

Can you post the FBX or HIP?

Edited October 12, 2018 by Atom

[PJM33]

Yah I ran the script and it did correctly add a new material node to each geometry object that seems to have a name that points to a Redshift material. 

But half of the geometry in the viewport is missing the materials and I can't get any of it to render. 

Am I doing something wrong? Any help would be greatly appreciated. 

 

Pierre

 

ContainerShipv01.hiplcContainerShipv01.hiplc

Here is the file. 

 

[Atom]

The viewport is another issue and is not addressed by this script. This script is supposed to help with rendering and setting up the Redshift materials automatically.

I looked over the scene and noticed that a lot of the FBX materials are all colored grey. So the script had nothing to detect and makes all materials the same color. That might be what you are experiencing when you mention you can't get any of it to render, a white render.

I also noticed that there are maps associated with the FBX materials. I altered the code to examine the texture filepath for map1 of the FBX shader. If the length of that field is non-zero then I fetch that value and pass it along to the Redshift texture sampler node. This ends up with relative texturing in place. Something like...$HIP/Downloads/Flags_diff.png

 

This means you need to create a Downloads folder inside your HIP folder and copy all the texture maps for the FBX into that folder. Then Redshift will detect and display them in the render.

If you are going to try the new script, try it in a clean scene. Import your FBX then run the script.

 

Edited October 14, 2018 by Atom

[Delda]

Hello Atom,

First of all, thank you for your script!

I tried it and but could not get it to work properly. I imported an FBX into Houdini 17, and ran the script. When diving inside the FBX subnet and then inside one of the geometry objects, I could see a new material node being added to the end of the network. The script also generated a buntch of Redshift Network at the /shop level. 

However, when I tried to render it using Redshift, the materials are completely missing. Can you tell what I did wrong?

I herewith attach the hip file if you feel inclined to examine it.

Thanks again!

 

 

Test_FBX.hipnc

[Atom]

I don't see any evidence in the FBX subnet, that you have successfully run the script. I do see that the materials were generated. Did you, perhaps, throw away the FBX Subnet then reimport it? Once you have run the script, you are locked into that version of the FBX subnet. (or run the script again)

When you have successfully run the script you will see two material nodes inside your object file. One has the PINK render flag set and will point to the Redshift version of the materials. The original one will have the BLUE display flag set and point to the FBX materials.

Edited April 27, 2019 by Atom

[Delda]

I retried again but could not get the result you said. I wonder whether this has something to do with the path being relative or absolute.

In fact, the model is originally made in SketchUp which you can download from the following webpage. When exporting to FBX using SketchUp, the process created relative path that Houdini fails to interpret correctly. (However, the textures of the exported FBX could be rendered properly in Maya.)

https://3dwarehouse.sketchup.com/model/1efbe858d6b4680c63995fb119e59971/St-Josephs-Oratory-Montreal

After a bit of research on the internet, I found out a solution that is to open the Sketup file in 3dsMax and export it as FBX. 3dsMax creates absolute path for the textures that can be read properly by Houdini.

 

So I have two versions of FBX of the same model, the one I tried yesterday was the one with absolute path exported from 3dsMax. When importing FBX into Houdini, there are two options under File Path: Keep Absolute or Convert to Relative. I tried both options with my two FBXs. No matter what version and what option I selected, the result is always the same: The script generated materials in /shop, but no second material node with PINK render flag in the geometry.

 

I have also tried exporting an OBJ instead and running your scripts 'MTL to Redshift Material', which did generate materials in /shop but the textures are not applied in the Redshift renderer. Your script 'Rewrite OBJ and MTL files' did nothing as all the texture names start with _auto_. This does not seem to be a naming issue with 3dsMax.

 

I herewith attatch the two FBXs I have. Can it be an issue with the model itself?

 

Thank you Atom for your prompt reply! It is much appreciated.

Oratory_absolute.fbx

Oratory_relative.fbx

Edited April 27, 2019 by Delda

[Atom]

The script works on both version of the FBX, for me. By "workling," I mean that it runs and makes materials and sets the pink flag on all the geo objects inside the FBX subnet.

It seems like the absolute version pulls JPG images from the .FBX file. Redshift can't do this yet, but the script reliably transferred that path to the new shaders, which fail to load the image, producing an all white render.

The relative version, however, produce a texture path like this, in the texture filename field.

Oratory/_auto_7.jpg

This means, if you save your hip file and create a folder named "Oratory" in the same folder where the hip file resides, and copy the JPG texture to that location, Redshift should be able to load it.

 

I did not realize an FBX could contain textures, like a ZIP file. I wonder if there is anyway to extract them?

Edited April 27, 2019 by Atom

[Delda]

I did as you said, creating a texture folder named 'Oratory' in the same folder where the hip file is saved. Then I double checked --

 

For the absolute version (no matter opting for Keep Absolute or Convert to Relative), the texture path in the RS Texture filename field after running the script is something like this:

Z:\Groupes-cours\NAND211-H19-N01\Partage\Hyperbolic-Space\PROJETS_3D\HOUDINI\Oratory.fbm/texture19.jpg

The script created a folder /Oratory.fbm on my drive which contains the texture maps. Notice that their names no longer start with _auto_. The slash after the folder /Oratory.fbm is a backslash whereas those after this folder are forward slashes. I guess this is because I am running Houdini on Windows 10. I changed a number of backslash to forward slash, but still could not see any texture being rendered by Redshift.

 

As for the relative version, the Filename is of the form Oratory/_auto_52.jpg, but when I clicked on the arrow next to the path, it sometimes points to the correct /Oratory folder, some other times it points to a non-existing folder whose name is of the form:

Z:/Groupes-cours/NAND211-H19-N01/Partage/Hyperbolic-Space/PROJETS_3D/HOUDINI/Oratory/Oratory/Oratory

The number of times the string /Oratory repeats varies. 

For this version of FBX, Redshift failed to render the textures too.

 

I guess not having the second material node with the pink render flag is what makes Redshift incapable of loading materials properly, yet I cannot think of any reason why the script should generate it for you but not for me ...

I am running the Education Edition of Houdini 17.0459 and Redshift with educational license on Windows 10 in school.

At home I have Houdini Apprentice 17.5.229 and Redshift demo on Mac OS 10.14.4. Each time I run the script, an error message always pops up, saying:

Traceback (most recent call last):
  File "FBX importer", line 235, in <module>
  File "FBX importer", line 211, in groupByFBXMaterials
  File "FBX importer", line 16, in createRedshiftImageMapMaterial
  File "/Applications/Houdini/Houdini17.5.229/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/houpythonportion/ui.py", line 850, in decorator
    return func(*args, **kwargs)
  File "/Applications/Houdini/Houdini17.5.229/Frameworks/Houdini.framework/Versions/Current/Resources/houdini/python2.7libs/hou.py", line 8510, in createNode
    return _hou.Node_createNode(*args, **kwargs)
OperationFailed: The attempted operation failed.
Invalid node type name

 

Edited April 28, 2019 by Delda

[Butachan]

@Atom Man, Thank you very much for this script! I was just about to embark in figuring out how to do this for Kitbash objects in FBX format. This will definetely solve my problems or point me in the right direction.

PS: I like the fact that you call 235 lines of codes a "short script" .  I am almost fearful to see your heavy ones 

 

Nico.

[Butachan]

Hi Everyone, I "expanded" and changed the solution by Atom and Delda a bit

The script now expect you to select the SHOP network node created by the fbx importer. It then creates MAT network at obj level with the same name as the selected SHOP network node and then I use ATOM function to create the redshift shaders inside it.

I puposely do not append the 'rs_' to the name of the material so it is easy to replace the original material by simply changing base directory of where the object is searching for the materials.

Basically the same materials name will reside in a different container.

 

The reason why I took out the part of the script that appended the new materials to the object was because I thought that as it was written it may have problem with the structure in which my FBX where built. Basically multiple materials adresses inside a single material SOP.

Anyway, As @Atom script was very helpful I thought in adding my bit by sharing the adaptation here and hoping the community find it useful also.

 

Nico.

fbxMat2RSMat.py