[H13, Helga] Crafting a solid, scalable Lighting/Shading Pipeline

[timmwagener]

Hey guys,

 

i thought that this topic is probably too large and specialized for the WIP thread, so it probably fits in here better.

This thread is about our attempt to design a well rounded lighting and shading pipeline for our animated

full cg diploma movie with the working title "helga". (A task that is up very soon...)

For more infos about it, please have a look at the WIP thread.

 

Transitioning from Maya/Vray/Arnold/MentalRay

---

Additionally this thread might become a good ressource for people that are very familiar with other rendering

pipelines (like the classic Maya/VRay pipe) for example and want to try something different, but struggle

to find equivalents to some of their old tested&true workflows.

 

This is basically a huge motivation for this thread as well

 

1. Shading Assets

---

What we plan to do is deploy our shaders as Shading HDAs. They are planned to be Subnets that contain all the materials needed.

You create them in a seperate file and everybody uses them in their shots. (Automatic update for HDAs enabled).

Assignment on the geometry is done as a "local edit" on the geo per shot instead of having an object merge node into your asset and assign it internally.

 

We had our shading assets with object merge nodes and internal assignment of materials so far, but found that to be unpleasing

when light-linking and takes are added to the mix. What is the common practice for shading Assets/HDAs in Houdini?

 

The goal is to not have takes (renderlayers) for different AOV passes, only for different elements (like characters, foreground, midground etc.).

 

An issue we have with the new setup is that Houdini often complains about the "F" variable of the surface model (Image attached at the bottom).

 

 

2. Global standardized additional AOVs

---

Im not talking about light passes here, since those are standardized. I'm thinking about AmbOcc., world position, wide/small fresnel passes that can be turned on/off

globally for all Helga shaders just like you would turn on/off RenderElements in VRay for example. I understand that Houdini gives us the freedom to care about this

ourselfes, so how would we do this?

Currently i envision a "standard_additional_AOVs" HDA embedded in our Shading HDAs that holds all the parameters and exports.

Additionally a little Python UI would parse your scene for HDAs of this type and allow you to toggle/adjust them globally.

You can then add them as image planes on your Mantra node (or maybe this happens automatically through the Python UI on toggle).

 

Nested HDAs update recusively right? So when you have a newer version of an HDA embedded in another HDA it will update!?

 

3. Decoupling Displacement and Shading

---

Can we separate the displacement from the shader nodes (but keep it at rendertime), just like VRay displacement sets do in Maya ?. The reason is, that we have 

displacement on almost every object (at least thats how it is at the moment...). It would be nice to be able to do global material overrides for the whole scene

without loosing the displacement.

 

4. Blend materials

---

What is the common practice for Blend materials with Houdini?

 

5. Alembic / packed primitives / Houdini Geometry / Crazy Highpoly Geo

---

How do you tend to bring in your Alembic files? As far as i understand the docs, when you bring in an Alembic Geo in H13 (which we are using)

as Alembic Delayed Load Archive it is super efficient. As we can solve our props really highpoly with Photoscan to capture tiny surface detail,

this is def. an option if it doesnt blow our workflow.

 

So in general, is bringing in Alembics in Houdini as ADLPs the same as having VRay or Arnold

Proxy nodes in your scene and loading highpoly geo at rendertime ? Can it take the same polycounts? (10million upwards per asset approx. x 30 assets...)

 

Will we put crazy traffic on the renderfarm network when we do this since all the geo data needs to be loaded at rendertime?

 

The alternative is, as we do it right now, to have moderate polycounts on all geometries and displace them with the luminance of

the colortexture.

 

6.Houdini LUT format issues

---

As we manage the color through our entire pipeline with OCIO, we wanted to try and use the Sony Pictures sRGB LUT for the show. (It has a slightly more contrasty curve than srgb,

which is ment to account for the dynamic loss of displays like monitors vs reality). However i couldnt bake a LUT out in a format that Houdini understand (.lut and .itx) that works.

Somehow the highlights are always clipped. The LUT gives the same transformation in MPlay, HouRenderview and Nuke. When i bake the same transform out as .csp (cinespace LUT)

for example, it looks right in Nuke....but Houdini cant read it.

Thats a minor point though, standard sRGB will be equally fine.

 

---

Pheww, that was a lot. Thanks for your attention, and i'm curious about your thoughts

Edited August 26, 2014 by timmwagener

[michael]

you may want to steal some ideas from Katana...

what I would do is this:

HDA - characterA (model, rig), with 'default' material assignments, you can leave the material assignment nodes as 'editable' in the HDA

HDA - shadersCharacterA

run a versioned script per shot to edit the material assignment nodes if needed and any other shot specific requirements

this way you can make changes quickly in the script and track them with a version #

[timmwagener]

you may want to steal some ideas from Katana...

 

Havent looked at it yet since its Linux only. (Although we have licenses at University i think). But one of our Lighting TDs just

came back from MPC where he worked with Katana i guess

 

HDA - characterA (model, rig), with 'default' material assignments, you can leave the material assignment nodes as 'editable' in the HDA

HDA - shadersCharacterA

run a versioned script per shot to edit the material assignment nodes if needed and any other shot specific requirements

this way you can make changes quickly in the script and track them with a version #

 

We are running entirely on Alembic for cameras and geo. But what you are saying is basically to make the assignment of shaders by script, right?

This makes sense and is exactly how we did it usually with Maya and shading asset references.

[michael]

go to the foundry site and grab the Katana user guide and have a quick read - it will give you some good descriptions of how many Houdni users have been managing large projects since HDAs were developed...

basically Katana doesn't change anything in the scene, instead it creates a 'recipe' - a list of edits, changes, assignments etc...that's what the file is...then you can make templates that have all these things inside and then (ideally) you just load different cameras and animation geometry and click 'render'

you can do something VERY similar in Houdini...

[illuMinator]

Hey guys!

 

I am one of the TDs on the project and like Timm said, I recently had the pleasure to work with Katana. Unfortunately not really long enough to understand it in depth yet. Michael, I don't exactly know what you mean for us to do like in Katana? Are you referring to lookfiles?

 

Concerning the shader assignment, you are basically saying to have the assignment already set up in the digital asset and only change them per shot, if we need to?

[michael]

yeah - the asset will already be 'looking' for it's shaders...and as soon as it instantiates into the scene file it will find the shader HDAs and have a valid assignment

as for Katana and lookfiles - yes, just the concept of having

assets (character_001) + a series of operations ON those assets (scriptedFile_0001_v4.scr) = Shot 001

assets (character_001) + a series of operations ON those assets (scriptedFile_0002_v18.scr)= Shot 002

 

your per shot edits have no effect on the assets themsevles...just on those assets when they are INSIDE that specific shot.

 

you can script just about anything out:

animation

changes to shaders

lighting setups (better with lighting_rig assets)

comp setups

etc

[dennis.albus]

Hi Timm,

sometimes it's a real struggle to get all the different aspects of Houdini to work nicely together to get a coherent lighting and shading workflow. Here are some answers to your questions and some more suggestions:

 

Nested HDAs only update if the asset name internally is the same. Eg. The namespace, the name and the version need to be the same. If you update the asset helga::wood::1.5 and save it again as helga::wood::1.5 it will update if it is nested in another HDA. However if you up the version to 1.6 the version of the nested asset is not automatically updated. You can write a simple script for this purpose.

 

Be careful with Cameras from alembics as they sometimes have weird inherited scale from maya which scales the light intensity when rendering (experienced this a couple of times now). We have built a camera extractor HDA which uses CHOPS to transfer all the important parameters from the alembic cam to a fresh Houdini cam without weird transformations. Works perfectly.

 

For splitting the surface and the displacement shaders you can have them separate and plug them through a switch nod into an output node. I attached a scene to show you what I mean. Alternatively you can simply add the surface shader and displacement shader separately to your objects. you don't have to use a single material which contains everything. This way you can easily override the shaders.

 

That's it for now. Maybe I can add some more during the weekend.

 

Cheers,

Dennis

shader_setup.hip

Edited August 29, 2014 by dennis.albus

[timmwagener]

Hi Dennis,

Thanks for your effort!

 

Be careful with Cameras from alembics as they sometimes have weird inherited scale from maya which scales the light intensity when rendering (experienced this a couple of times now). We have built a camera extractor HDA which uses CHOPS to transfer all the important parameters from the alembic cam to a fresh Houdini cam without weird transformations. Works perfectly.

 

I'm working on the Alembic export process right now. The goal is to apply all fixes and adjustments (like simulation Preroll etc.) implicitly during the export, independent from the animators source scene.

If you can pinpoint whats causing the camera issue on the Maya side (like animation on scale attrs...!?), we might already tackle that during export. If not, an HDA seems just fine, as long as our single data exchange format can stay Alembic and we dont need to employ a different format for each entity.

 

That's it for now. Maybe I can add some more during the weekend.

 

Your effort is really appreciated, but dont ruin your weekend

[dennis.albus]

I'm not completely sure what causes the problem.

The first time I experienced it the scene suddenly rendered black. While searching for the problem it turned out that I could either scale the light intensities by 100 or scale the camera IN PLACE by a factor of 100 (a null in the same position as the cam parented and scaled by 100). The problem seemed easy as 100 is a pretty common scaling factor when we are going from Maya to Houdini and it seemed like a simple export problem. However the next cam I got needed a scaling factor of 6.8 which I cannot explain until now. To solve the problem without much investigation we decided to just handle it with the HDA which works fine for now.

 

However it is possible that this might never be a problem for you. We did a lot of projects before and this was never a problem.

Edited August 29, 2014 by dennis.albus