Batch render objects

[konstantin magnus]

I am working on a script to render all OBJ-files from a folder separatly in the same scene.

How can I make a python script wait for a render to finish, so it can then load the next OBJ file? (I am not looking for a frame-based solution.)

Edited October 1, 2018 by konstantin magnus

[StepbyStepVFX]

Hi, I don’t know your constraints, but at first sight I would :

- put all my objects in the scene, into one object container at /obj level, with many file nodes inside : using a Python script to scan the directory, and loop over each object file to create a file node into an object.

- I would connect them all to a switch node (can be done in Python as well)

- then I would create a null, with an integer parameter, and reference this parameter on the switch node.

- then I would create a Mantra node (or any other ROP)

- use a good old Wedge node that point to that Mantra / ROP node, and wedge the paramter that is on the Null above. Wedge is sufficiently intelligent to wait for each render to terminate before triggering the next rendering

You could even directly wedge the switch input from the switch node, but I like to put parameters on a null :-)

And on top of that, I think the Wedge node can send different renders simultaneously if you have a renderfarm (not sure, but I think I have read that somewhere).

Hope that helps you

Edited October 1, 2018 by StepbyStepVFX

[konstantin magnus]

Hi JO, thanks for helping me! I thought about using a wedge SOP, as well, but what I really want to create is a render machine, that is sucking in upcoming OBJs, integrates them in a given scene and renders them out immediately. It can potentially be hundreds or thousands of different meshes, so I would only load them one after the other.

Thats why I am rather thinking about a python script that manages all this:

• Create a list of mesh files from a directory, and iterate through these steps:

1. Assign one of the meshes to file node
2. Press render button on mantra ROP
3. Wait for the rendering to finish, and go back to step 1
4. (until the list of mesh files is depleted)

How can I make a python script wait until the render process is finished? And where would I put this script (post-render script)?

 

[StepbyStepVFX]

I am sticking to my first idea because I have no clue how to make Python wait for « token » saying render is finished (maybe by asking Mantra to output a log - with appropriate verbosity level - and read continuously the log file until you find the 100%... I have seen the log files on a render farm, so there must be a way to ask Mantra to output it... then it is a matter of parsing the file and finding strings :-)

But reverting to my first idea : you could use Python to scan your directory, list the obj files, and creates points with a string attrib pointing to the file. Then use the wedge trick to delete all the points but one (the one with the wedge parameter) and use point instancing to render ....

[konstantin magnus]

4 hours ago, StepbyStepVFX said:

you could use Python to scan your directory, list the obj files, and creates points with a string attrib pointing to the file. Then use the wedge trick to delete all the points but one (the one with the wedge parameter) and use point instancing to render ....

Clever workflow!

But how do I assign the file string to the instance node?

(I found 'Delayed Load Shader' in Houdini´s documentation and 'Packed Disk Primitives' here: http://www.toadstorm.com/blog/?p=493, though I do hope there is a more direct way to swap out files).

Edited October 2, 2018 by konstantin magnus

[StepbyStepVFX]

I would have to look again to the old Sidefx video about instancing, but I think there is this point attribute called « instancefile » or « instancepath » instead of « instance » (which is used when objects are at /obj level). I don’t remember exaclty its name... But I think it must point to a bgeo file, as well.

The biggest problem with that is shader management : they need to be all into the hip file with export all shaders in ifd selected on the Mantra node. But I had troubles instancing OTL that contained their own

So I let you the joy of trials and errors :-) 

Let us know if that works

EDIT : another idea to avoid instancing :

- you do the thing with points that have a path attribute pointing to an object file on disk

- you delete all but one using a parameter that you are going to wedge

- you have one single file node whose path is evaluating the parameter of the point 0 of your points stream (after the delete where you keep one point based on the wedged parm). I think ‘point(´´ obj/blablabla /delete1’´, « path »,0)´ with backticks could work, isn’t it ?

hope that helps !

Edited October 3, 2018 by StepbyStepVFX

[konstantin magnus]

The instancing trick works! I can now render out all my meshes with one click (also added a lens shader to get more perspectives in the same image):

Unfortunately the wedge ROP names files with seed numbers: "Mesh__wedge_instance_point_8.000000.png", where I would much rather like to get the original file name like "HANNIBAL_obj.png". Changing file names in the file node does not work, because backticks are apparently not accepted there. At least when I enter

`point("../instance_point/", 0, "path", 0)`

the file node errors out saying 'Unable to read file "0"'. I also tried setting the file node´s path from another python SOP:

node_file = hou.node('/obj/mesh/import_mesh')
point = geo.iterPoints()[0]
path = point.attribValue('path')
node_file.parm('file').set(path)

.. which works the first time, but sadly the wedge node does not update it. Then I tried forced cooking via Python or laying out dummy wedge channels to all nodes that need updating to no avail.

 

So is there any way to get the *.OBJ file names out of Wedge ROPs? Or any alternative routes to turn a folder full of meshes into renderings?

Edited October 7, 2018 by konstantin magnus

[Atom]

What about something like this. Place a python script in the Mantra Pre-Render event and simply re-write the File node and the Mantra output node filename. The frame number acts as an index in to the list of OBJ files found in the folder so match the frame end to the number of OBJ files in the supplied folder. Render the range sequence and on each frame the new filename will load and be rendered to a like named image file.

# Run in Pre-Render event of ROP.
import hou, os, re
    
def returnFilesLike(passedFolderName, passedFileExtension = ".obj"):
    result = []
    for file in os.listdir(passedFolderName):
        if file.endswith(passedFileExtension):
            result.append(os.path.join(passedFolderName,file))
    return result


def assignFolderOBJToFileNode(passedPath, passedType):
        lst_files = returnFilesLike(passedPath, passedType)
        if len(lst_files):
                f = hou.frame()
                for i,file in enumerate(lst_files):
                        if (i+1)==f:
                                # Re-write file name.
                                file_node = hou.node('/obj/geo_file_to_render/file1')
                                file_node.parm('file').set(file)
                                
                                # Re-write Mantra output ROP filename.
                                mantra_node = hou.pwd() 
                                filename, file_extension = os.path.splitext(file)
                                s = "%s.exr" % filename
                                mantra_node.parm('vm_picture').set(s)
                                print file, s

# Point to your folder.
folder_path = r"/media/banedesh/Storage/Documents/Models/Redshift/grass"
assignFolderOBJToFileNode (folder_path,".obj")

 

ap_render_folder_of_OBJ_like_OBJ_name.hiplc

Edited October 7, 2018 by Atom

[konstantin magnus]

I was initially looking for a non-framebased solution, but this seems more straight-forward. Thank you!

[konstantin magnus]

Based on @Atom´s post I am currently writing my own render script (see below). But no matter where I put it (pre-render or pre-frame), the file node is only switching to the next mesh when I manually scrub through the timeline and click 'Render' in the render view. When I hit 'Render to Disk' however, Houdini creates only images showing the same object.

Any ideas why the file node does not get updated here?

 

# PRE-FRAME RENDER SCRIPT
import hou, os, re

# NODES
node_import = hou.node('/obj/Mesh/import_file')
node_render = hou.pwd()

# INPUTS
filetype_input = '.obj'
filetype_output = '.png'
path_models = r'C:\Users\konstantin\Documents\00_Eingang\Meshes'
path_output = r'C:\Users\konstantin\Documents\00_Eingang\Renderings'

# FUNCTIONS
def search_files(root, type):
    print('Searching all {} files in {}'.format(type, root))
    list_files = []
    for path, subdirs, files in os.walk(root):
        for name in files:
            if name.endswith(type):
                path_full = os.path.join(path, name)
                list_files.append(path_full)
                #print('Found {}'.format(path_full))
    return list_files

def extract_filename(path):
    filename = os.path.basename(path)
    print('Extracting filename: {}'.format(filename))
    return filename

def set_import_file(node, path):
    print('Setting import file on {} to {}'.format(node, path))
    node.parm('file').set(path)

def set_render_file(node, path, name, type):
    path_full = path + '/' + name + type
    path_full = os.path.normpath(path_full)
    print('Setting file name on {} to {}'.format(node, path_full))
    node.parm('vm_picture').set(path_full)

# PROCESS
file_list = search_files(path_models, filetype_input)

for i, file_path in enumerate(file_list):
    if i + 1 == hou.frame():
        file_name = extract_filename(file_path) + '_' + str(i)
        set_import_file(node_import, file_path)
        set_render_file(node_render, path_output, file_name, filetype_output)

 

Edited October 8, 2018 by konstantin magnus

[StepbyStepVFX]

Hey, I know I will look stubborn, but check this file, this should do what you want : it gives your image a name that contains the name of the object you are rendering (using splitpath). You'll have to adapt it as I don't know how you stored the data, but the file is easy to understand.

Just use the same wrangle and the null whose purpose is to capture the name of your obj, in your Instance node. The parameter of the null I have added is then referenced in your Mantra node, in the name of the saved image. All of that before using the Wedge that controls everything.

Hope that will solve you problem for good :-)

wedgeRenderObj.hip

Edited October 8, 2018 by StepbyStepVFX

[Atom]

You are right, hou.frame() is not updated by the Mantra ROP when you render an image sequence...weird.

What I did discover, however, is that hou.frame() is updated when you are saving out a FileCache sequence. So you can just place your code into a FileCache ROP, it does not have to be the Mantra rop. The only thing I added was to click the Mantra node render button after setting the filenames.

 

The progress is funky on this. The FileCache exports it's junk geometry and issues the python script. The python script clicks Mantra render button but does not wait for it to finish. It seems like several Mantra processes start running at once and kick the fan speed up real fast. But if you just wait it out, and monitor your OBJ/IMG folder you will find that the mantra node is making a thumbnail for each object in the folder.

# PRE-FRAME RENDER SCRIPT
import hou, os, re

# NODES
node_import = hou.node('/obj/geo_file_to_render/file1')
node_render = hou.node('/out/mantra1') #hou.pwd()

# INPUTS
filetype_input = '.obj'
filetype_output = '.png'
path_models = r"F:\Keep\Models\Plants\Grasses"
path_output = r"F:\Keep\Models\Plants\Grasses"

# FUNCTIONS
def search_files(root, type):
    print('Searching all {} files in {}'.format(type, root))
    list_files = []
    for path, subdirs, files in os.walk(root):
        for name in files:
            if name.endswith(type):
                path_full = os.path.join(path, name)
                list_files.append(path_full)
                #print('Found {}'.format(path_full))
    return list_files

def extract_filename(path):
    filename = os.path.basename(path)
    print('Extracting filename: {}'.format(filename))
    return filename

def set_import_file(node, path):
    print('Setting import file on {} to {}'.format(node, path))
    node.parm('file').set(path)

def set_render_file(node, path, name, type):
    path_full = path + '/' + name + type
    path_full = os.path.normpath(path_full)
    print('Setting file name on {} to {}'.format(node, path_full))
    node.parm('vm_picture').set(path_full)

# PROCESS
file_list = search_files(path_models, filetype_input)

for i, file_path in enumerate(file_list):
    if (i + 1) == hou.frame():
        file_name = extract_filename(file_path) + '_' + str(i)
        set_import_file(node_import, file_path)
        set_render_file(node_render, path_output, file_name, filetype_output)
        hou.parm('/out/mantra1/execute').pressButton()

 

ap_render_folder_of_OBJ_like_OBJ_name.hiplc

Edited October 8, 2018 by Atom

[Atom]

Hmm...you know what. On a whim I tried replacing the node_render variable using the full path instead of hou.pwd().

This seems to fix the original problem. hou.frame() seems to work again.

node_render = hou.node('/out/mantra1')  #hou.pwd()

What if when Mantra moves to the next frame it presents a different hou.pwd(). So when you write the filename using that variable, it get's set on something that has fallen out of scope? By writing to the full path you make sure you are targeting the render node filename..?

Edited October 8, 2018 by Atom

[giovanni1990]

HI guys!

I found this topic very helpful and inspiring, thanks a lot. 

I used it to create a rendr machine, selecting files form a folder and render them with Redshift. For each object in the folder 3 different cameras and looks fine.

The only problem I have is that if I set in my ROP node to render just 1 frame (for each object of course), it doesn't work, since it's not waiting for the current render job to finish to start with the next one. No matters if the next render is a different camera for the same object, or a totaly different object with the same camera.

It ONLY works if i set the frame range in the ROP like from frame 1 to frame 2. Works perfectly but of course It doubles my render time since it's rendering two times the same image....

do you have any idea on what can be the problem? 

 

Thanx a lot

[giovanni1990]

I solved it. using the wedge node from the OUT context seem to give me that error 'cause of a mismatch of comunication between the cpu and the gpu ( since I'm using RS).

Instead using it from the TASKS contest and using 2 different local scheduler is enough to tell the gpu to wait. Great