Transfer HDA animation

[kiryha]

I have a Digital Asset of my character. It is animated in the animation scene. Is it possible to transfer animation to another scene with the same asset with standard Houdini tools?

[LaidlawFX]

There is no standard tools per say.

But you can copy and paste the nodes/HDA from one scene to another from two open sessions they use the same copy buffer, or merge the hips from the file drop down.

If you want to invest time you can script it, or make the several different nodal methods to copy the info/hda.

 

[kiryha]

If there are no standard tools than I will have to create my own...

At a high level, how should I transfer animation? Export-import animation curves of HDA as well as parameter values?

[LaidlawFX]

So an aside that may have found more of what you want...

There are the Python Panels: Autorigs, Pose Library, and Character Picker that ship with Houdini.

These have been updated by I believe Michael Goldfarb. Here as an example: https://www.sidefx.com/tutorials/autorigging-masterclass/ So it may be worth not listening to me, and looking into those and getting in touch with SideFX, before I take you down a more complex path, lol. Sorry these are new tools I do not use in production.

At the heart of it you would want a library where you could save your different animations too. So Walk, Run, Sit, etc... You would need to be able to read and to write from these files to the parameter pane. You may want to split up the rigs export to different components even like upper and lower, or other component pairings.

You can do this with say FBX files, chops with chan files, etc. Storing the animation with FBX may be good if your animators liek to use MotionBuilder, Maya and other programs. I would say your most studio robust and flexible would be encoding all with python. You can even go into the realm of making your own Python Panel in Houdini to help manage the files on disk and help import, including nifty images/gifs of the the different exports. I believe the python panels mentioned above can be used as examples if not used directly to get you started. 

 

 

[kiryha]

Hi, Ben!
Exactly, I used Autorig tool and learn how to use it with masterclass you mentioned!

Now I need to understand what options do I have to transfer animation data from animation scenes to rendering scenes to build pipeline tools for that. So, as far as I understand I have such alternatives:
1) Export-import (copy-paste) animated assets themselves. Not very reliable and flexible way, I do not consider it as a possible solution.
2) Export-import caches.
3) Export-import animation channels.

The last two solutions are fine (personally I would prefer caches cos you don't have redundant data like rigging nodes in the render scene) but choosing between them rise another question: how to work with materials. And materials workflow is another big question from the list of all pipeline questions I need to figure out. Seems like common practice is to have the same HDA (which holds everything it needs in all production steps: rigging, materials etc) which could be loaded in animation and rendering scene and transfer animation data only between this departments. In the case of geometry caches, the material assignment system needs to be developed.

PS.  I am not using any other software, pure Houdini. 

[LaidlawFX]

It's good to know you're pipeline is just in Houdini.

* IMHO, you can ignore this part, just don't build yourself too much into a single software centric pipe. One software generally won't be the solution for everything, and production every-software has it's strengths. For instance Shotgun, Nuke and Houdini all go hand in hand.

So if you are doing strictly shot based work from anim to rendering, like a film or commercial pipe. Plus if you are not working on the part of the pipe to help your animators read from a library of animation cycles you can simplify it pretty quickly. Also not working on something to help FX team then it's pretty simple.

For Lighting/Rendering just go with geometry caches. This way the lighting artist/rendering person only needs to read the caches from each respective department. On your character rigs you will just need a render button, that will export your geometry sequence .bgeo.sc to a project/sequence/shot#/department/asset/version structure or something similar, you can get real fancy with the file structures. For instance, for relative common asset just replace common in the corresponding hierarchy, and mirror of authoring scenes, and machine generated content. If you want you can even include an .xml/jsn for meta data on import, or include it in the details attributes for custom tags of information, static asset, frame range viable, render option etc... This way your Lighting artist can just load all assets from the same hierarchy. By having all the geometry cache in the same known structure it becomes easier with out a program like Shotgun to manage the asset dependencies to auto load all assets on launch of the .hip file based on file locations. 

Sometimes it's good to export the animation channels out on the rig/hda, as you can use the same animation caching method that your animators will use. Save you from developing an extra system. But you are going to need a geometry cache system, that is the most common production tool across all studios. Also for FX artist, it is sometimes really needed to have the entire rig for simulations and FX work. Character TDs doing grooming, cloth, or rag dolls for instance would need them depending on the work flow. If you save your T-pose in an easy to access mode on your HDAs this becomes easier, but since DOPs can inherit object transforms easier it can be a toss up depending on your artist. 

For materials yes you can keep them in the rig. The thing to remember is they are only a string attribute on the primitive, so you can bend and twist this string to your hearts content. 

It also really depends on the complexity of your production, or are you building a pipeline for a studio to do hundreds of projects? Are you doing a simple commercial no change in materials for the whole sequence, or are you doing a fashion commercial where every shot is the same pirouetting ballerina rocking different clothing? Also are you shading an entire crowd or just one character. You do not want to have hundreds of the same material loaded in a scene in that case. You would want to have a load detector for a common material area, if this material has been loaded do not reload it. All thousand characters would use the same material, this will certainly render simpler and keep your scene file manageable. Also do you need per shot flexibility on adjusting the materials, especially for the light artist? The possibilities for mantra and comp have gotten so many amazing options than two decades ago you may not need any of those if you have a competent compositor and one person to set up the shot. 

Hope it helps, there are a lot of questions to reflect on scope, scale, desired functionality, flexibility, time and resources... So not really one best answer with out those variables.

[kiryha]

Thanks, Ben for that information, really useful!

When I sad pure Houdini I meant major players in 3D area only: no Maya, Blender or 3DS Max, etc. Sculpting and painting tools like Z-Brush and Mari are in implementation plans at a later stage. Nuke and Shotgun are the part of the pipeline already, however, I tend to render final images from 3D and do in comp a very few things only. Also, I am thinking to implement Shotgun as an optional choice cos it's quite expensive and not everybody has an account (and as far as I know there are no free alternatives to Shotgun with Python API) but its quite tricky to manage project data without such system. When I start playing with Houdini I realize that it could handle all 3D operations quite well (opposite to a common opinion that Houdini used mainly for FX), this inspires me to switch from Maya completely and build a Houdini based pipeline. I am doing a personal project in Houdini (almost without a character animation but I still need this functionality to exchange data between different 3D stages) and pipeline developing is a part of this process. As a result, I wish to get the solid pipeline basement which could be scaled for next bigger and more complex projects with a bunch of artists involved. I know this is not the best idea trying to create an abstract pipeline for all possible cases, so I would say full CG animation film is the project I am aiming for (even if it`s very small and could be done with one artist). 

Materials... There are tons of possible options but I am thinking if it makes sense to use in Houdini the same solution I designed for Maya: you create a library of common materials (wood, grass, plastic, metal etc) and apply materials in rendering scenes and each asset has shading attributes (material name, textures names for different channels, individual tweaking parameters, like reflection straight). In such case, every asset in production consists of mashes combined by materials. So you have a very limited amount of shaders in the final scenes ready to be rendered but with an ability to tweak each asset shading individually in each shot if you need. I still don't like this idea to have hda in render scenes with all redundant information and individual shaders (when a lot of them could be shared across different assets) but my mind is opened.

So I will examine the geometry caches option (bgeo.sc) for now. My current question is: if I will load animated asset with File node into a rendering scene, how can I distinguish different parts of the geometry and apply different materials?

[LaidlawFX]

So yet again not a an extremely simple answer, so this will spiral out as you read through it more.

To keep it simple with the materials. You can just use groups/attributes to define the different parts of your character and then with the material sop you can define your materials per a region. Plus what I think you are after is the material overrides. You can define the overrides to the shader on the mesh. So you could define the diffuse map per each primitive. This can get a bit expensive with the file size on disk, especially for an animated sequence, but you could theoretically have one uber shader for the whole production, and everything is defined via the mesh. You can do this at the object level, too depending on how you split up your object. Finding the happy medium is key. For a one person project, and for standardized shading models this might work out really well.

An alternative that is similar and will not over use the material overrides, but still have all presets defined on your uber-shader, is by having packets of shaders within a material subnet HDA. So you could save all the presets shader for your character, for instance, all the clothing types in one material HDA subnet, that contains all possibilities. i.e. Ralph_Clothing.hda with the material subnet hda containing clothes/dirty clothes/clean clothes/pants

A more advanced setup would be on import of your object with a file wrapper on your HDA, to parse the geometry or an external saved file for subsequent required materials. You can do it similar to the above, or even by creating the uber shaders dynamically in python and applying the presets directly in the scene. By applying the presets in the scene, say if you have a lookdev department authoring the materials, you may have issues maintaining their updates. However, with even more pipeline framework you can dynamically refresh those once they are updated. Another alternative, in the simple case is that the lookdev department authors their materials as HDAs too. Then you get the benefit of version control built in. But this requires you to build a lot more complex naming convention system, for storing HDAs, etc.

There are also material stylesheets you can use. Jeff Wagner just did a few video's on those, but they may be more up your alley, a bit too much to explain in a forum post however.

 

[kiryha]

How to store materials if I wish to have a material library (ies) and use it (them) in render scenes?

First I was thinking about storing materials in HDA but I have some concerns. According to my method of structuring materials, there could be one library for all environments and props and each character will require its own library (because of a complexity and uniqueness of character assets its easier to manage unique materials on each character). If there are many characters in a project the HDA list in Houdini will become massive, and also materials would be mixed with other asset types (characters, props, etc). Imagine Digital Assets menu will contain the same amount of assets as "All"...

Maybe there is a way to structure HDA other than by using suffixes in HDA names (folders)? Or it's ok to have a huge list cos nobody will ever browse this list in the usual way and all assets creation and update will be scripted?

[LaidlawFX]

These are certainly some good reservations about storing the materials as HDAs. No matter what you will have a massive collection of files to stores these materials. These materials could be substance materials, or as jason/python file as other examples to apply the same logic thoughts.

As far as the shit loads of HDAs, you can store these appropriately in your HOUDINI_PATH or HOUDINI_OTLSCAN_PATH. This way you only load the HDAs that you need. These would be the separate load directorys you append on load. You can also import via python files from disk not already in those directories, which may be the best method. Houdini does not really care about how many thousands of HDAs you load, assuming you didn't build really bad HDAs that cook the world on load. Generally speaking you don't actually need that many HDAs for even a feature film. A couple hundred might be enough for all FXs and studio functions. A few dozen really matter in the grand scheme. As far as materials the whole concept is to have as common as pool as possible. You don't want a dozen ways to do car paint.

You can also manage hiding the HDAs via OPCustomize file. So you could load all the HDAs and ophide them all by default. Look through that file and you'll see a whole pile of tricks.

On creation of these HDAs, as you do not want to do them by hand, you can sort them by the tab-submenu paths, and update the OPcustomize, or any other corresponding HDA options, via python. This way they are organized better than by hand.

Honestly, in the case of materials required for objects you do not want the Lighter to ever have to look up the correct materials for the objects. So seeing them all like in "All" shouldn't be a real concern. Only OCD like me and you notice or do anything about it. I've worked for many studios where people just tab and go. So there will be a level of automation you will need to develop for your project to load them either way. Cycle back to the shotgun conversation that can help manage that. 

[kiryha]

9 minutes ago, LaidlawFX said:

You can also import via python files from disk not already in those directories, which may be the best method

You mean store material libraries as hip files and merge to a render scene? This is the way I am thinking to deal with materials now.
Or you mean store material libraries as HDA and install them (I have no idea how HDA importing works)?

12 minutes ago, LaidlawFX said:

Honestly, in the case of materials required for objects you do not want the Lighter to ever have to look up the correct materials for the objects

Definitely, materials would be assigned automatically in render scene via groups and @material attribute (which should be set during modeling or lookdev stage) after cache file is loaded:

[LaidlawFX]

 

19 hours ago, kiryha said:

You mean store material libraries as hip files and merge to a render scene? This is the way I am thinking to deal with materials now.
Or you mean store material libraries as HDA and install them (I have no idea how HDA importing works)?

Nah not merging hip files. That would be quite dirty. You would use python to instantiate the required node/hda and apply the appropriate presets. Whether with the SideFX system, or probably more likely your own. The current SideFX system leaves much to be desired as far as presets.

So for your materials, you only want a handful of studio HDA used by default for most lookdev. Generally only your FX artist should make their own shaders. You don't want to have different lighting models for each material it can get crazy quickly the more artist your team uses. 

Look at this talk on Houdini Pipelines. It will probably help a lot.

 

19 hours ago, kiryha said:

Definitely, materials would be assigned automatically in render scene via groups and @material attribute (which should be set during modeling or lookdev stage) after cache file is loaded:

You may be best off just using the stock shop_materialpath attribute, because you will need to bind the path and the material in the end to render in Houdini. You can avoid groups too in this way as you can not have two materials per a primitive. The groups are good if you need to reassign the materials, but that should only be done in lookdev. 

[kiryha]

I saw this video a couple of times, as well as some other Houdini pipeline. I feel now I need a more low-level explanation how can I store and assign materials (Mantra Principal shader mostly).

Say I have a cached character animation which I bring to a render scene as a File node. As far as I understand I will need a material library asset for this character and I know how could I create it and bring to my render scene. But how can I assign materials to a single mesh other than with groups in Material node? And how can I use shop_materialpath attribute, I did a quick search but did not get any a description of related workflow?

[LaidlawFX]

For shop_materialpath crack open the test rubber toy HDA that ships with Houdini. You will see how the material is applied to the geometry in the spreadsheet. You might want to watch some intro to lighting and rendering in Houdini. For the most part it has not changed in the last 10 years. The lighting nodes, mantra's ability, and the material networks have improved, but at the heart of it is the same pipeline. Hopefully in the future they will do a major update to the workflow * fingers crossed *.

There are not many documented Houdini Pipeline workflows around Lookdev as that part of Houdini pipelines is actually on the rare side. So Disney, DD, Sony will have their own custom workflows, but smaller commercial shops will just have a few Arnold or other lighters do it on the fly. The FX guys in those shops just need to rock off a few quick materials, and for those folks the basic are enough. What you are asking for is a bit more rare. But then again welcome to Houdini \o/

Generally speaking I never use the material Library, nor do most productions. You can actually hide that pane. It's the only place in Houdini that has a crazy wrapper for placing HDAs. You can actually just create a shelf that drops the materials with those presets so that it is available in the Tab Menu. At the end of the day, it's just the same few nodes with a few shelf like presets. Which when you have to do a full production pipeline you will need to make yourself. 

As far as the groups you will need to have your lookdev pipe do the assignment of materials. If that means importing the geometry with groups then do it, once the materials are assigned, then you can get rid of the groups unless you need to dynamically change the materials down the road. Groups used to be real bad a few versions back so most people have moved away for them for attributes use. The only reason they are not as bad anymore is because they behave exactly like attributes, except for the legacy menu options.

Lol, are you ready to tear your hair out yet, lol?

[kiryha]

1 hour ago, LaidlawFX said:

Lol, are you ready to tear your hair out yet, lol?

Ah, not even close!

 

1 hour ago, LaidlawFX said:

open the test rubber toy HDA that ships with Houdini

If I understand correctly... when I build my model (in modeling scene) and use material sop to apply materials (say from HDA in my scene) it creates a @shop_materialpath and record path to a material there. Now I can cache geometry (in production I will cache it from animation scene) If I will load geo cache to a render scene and create material HDA, Houdini will assign materials automatically.
It could be a basement for materials workflow... I test it and it works as I describe, everything is correct here, right? 

I am asking because sometimes things work when they should not. And it is even scarier than if things don`t work but they should... Don't want to build a workflow based on a wrong assumption.

Edited July 18, 2018 by kiryha

[LaidlawFX]

Nope what you are doing with @shop_materialpath is the fundamental basics that has not changed in a long time. Albeit we do not use the shop context as much anymore... now we use materials, but they won't just shut it off at some point when they make the switch. 

Manipulating this path is the key for material assignments. It's always an absolute path, so if you store your materials at an object level context like /OBJ/MATERIALS (or always in the material context which I don't suggest) as you did it becomes real easy to reassign the materials with basic string edits, and especially python. You could make a string attribute like "material" as you were doing before or "name" as is common for dop workflow(though consult with your fx team first) and then use that to help you reassign materials if you need later. Having two string based attributes is really cheap as it saves them in a dictionary, unlike unique groups that use integers. 

Since it's a dictionary of values you can just list the unique instances of those materials, and create a respective hda with python based on where that material should be stored. So /OBJ/MATERIALS/Sequence/... /OBJ/MATERIALS/Shot/... /OBJ/MATERIALS/Character/... /OBJ/MATERIALS/Common/... you can also use those names to create the correct HDA (like principled shader) and apply the correct preset to those HDAs now however you stored them. There are a lot of possibilities here, so what ever works best for your system. 

[kiryha]

Quote

it becomes real easy to reassign the materials with basic string edits, and especially python.

I did not get why I need to edit string if everything is working without modifications and additional attributes? I use the same material library HDA (/OBJ/MATERIALS) in modeling, animation and rendering scenes, the absolute path remains the same and materials automatically caught by the geometry from caches or material SOP in HDA.

Anyway, thank for your time, the information you provide is really useful!

Edited July 19, 2018 by kiryha

[LaidlawFX]

Once you get to the point where you are pulling library of assets, and modifying them for different sequences, shots and environments. Crowds, Cars, Fire hydrants, city blocks more standard Houdini bread and butter. 

Cool good luck!

[kiryha]

Ok, that makes sense. I am far away from such stuff for now.

I was thinking how to transfer cameras from animation to render. How its usually done, alembics, HDA, other?

[LaidlawFX]

lol. I would stick with alembic. It's a flat hierarchy. Once you import one you can see how the creation script modifies the default nodes. Then you can make an HDA from that imported files, that does the same, but all you need to change is the file it loads from disk. It's pretty obvious and interesting how it works. Then you can use the same alembic camera in all DCCs.