The relationship between the concept of a shader, a material and VEX

[odiwxe]

Hi, I've recently started learning Houdini, I've been surprised by the depth Houdini has and the freedom it offers, mire than anything I've been amazed by the SHOP context, I've always liked to make stuff lol more realistic in the final render but as a beginner i've been feeling confused by the terms,i can't understand the cancept of the word shader,some one said it's the way a pixel gets rendered but doesn't the material and texture determine how a pixel is rendered either?so what does a shader do?how can i feel the shader?for material i can feel it by seeing that for example a clay materials looks like that,renders like that etc but how can i visually see the shader in action?i want to know:

1- what's a shader?

2- what's a material?

3- what's a texture?

4- what's VEX and VOPs?

5- and finally what's the relationship between them?after these all i want a book or tutorial you suggest for an in depth houdini learning

 

thank you

Edited August 16, 2014 by odiwxe

[Potkonny]

- http://forums.odforce.net/topic/17105-short-and-sweet-op-centric-lessons/

See post #7 - "old school".

 

- http://www.sidefx.com/index.php?option=com_content&task=blogsection&id=14&Itemid=132

- http://www.sidefx.com/docs/houdini13.0/

- http://www.papicrunch.net/GC-houdb/?collec=0&model=list&filter=none&sort=idASC&p=6

- http://vimeo.com/channels/54102

- https://www.youtube.com/playlist?list=PLShEm1_z6_cyBOAKQSH2ck-enGSnovMjD

- http://forums.odforce.net/topic/20617-garmans-vizy-acky-houdini-intro-vol100-now-on-youtube/

- https://www.youtube.com/results?search_query=houdini+modeling+tutorial&page=1

- https://www.youtube.com/channel/UC0VQvtms-36ilCVgJQXEM_Q/videos

- http://www.youtube.com/user/Khermah/videos?view=0&sort=dd&tag_id=

- http://houdini.dreamerzstudio.net/

- http://lesterbanks.com/category/houdini/

- http://training.adrianlazar.com/free-houdini-training/

[fathom]

at the risk of oversimplifcation...

 

1- what's a shader?

 

a shader is an operation performed at render-time (typically for the current ray being rendered, but not always).  in houdini, there are a number of possible shaders: surface, displacement, lights, shadows, and others i can't remember off the top of my head.

 

2- what's a material?

 

in houdini, a material is a essentially a container for connecting a surface shader and a displacement shader for convenience purposes.

 

3- what's a texture?

 

in cg speak, a texture is an image (usually) that shaders use to assist in the shading process.  the most common use of textures is to project them onto your geometry and apply them as a color map.

 

4- what's VEX and VOPs?

 

vex is the custom language that shaders are written in.  vops is a graphical interface to assist in the creation of vex code for artists.

Edited August 17, 2014 by fathom

[edward]

1- what's a shader?

 

​The process of assigning the colour to a pixel by the renderer is called "shading". To do shading, it uses a "shader". The shader has various input parameters that allows for artistic control over what the shader does (like diffuse colour, roughness, etc.). In Houdini, I think of the bundle of specific parameters along with the shader, as "material". That is, many materials can make use of the same shader, but with just different settings. 

[odiwxe]

- http://forums.odforce.net/topic/17105-short-and-sweet-op-centric-lessons/

See post #7 - "old school".

 

- http://www.sidefx.com/index.php?option=com_content&task=blogsection&id=14&Itemid=132

- http://www.sidefx.com/docs/houdini13.0/

- http://www.papicrunch.net/GC-houdb/?collec=0&model=list&filter=none&sort=idASC&p=6

- http://vimeo.com/channels/54102

- https://www.youtube.com/playlist?list=PLShEm1_z6_cyBOAKQSH2ck-enGSnovMjD

- http://forums.odforce.net/topic/20617-garmans-vizy-acky-houdini-intro-vol100-now-on-youtube/

- https://www.youtube.com/results?search_query=houdini+modeling+tutorial&page=1

- https://www.youtube.com/channel/UC0VQvtms-36ilCVgJQXEM_Q/videos

- http://www.youtube.com/user/Khermah/videos?view=0&sort=dd&tag_id=

- http://houdini.dreamerzstudio.net/

- http://lesterbanks.com/category/houdini/

- http://training.adrianlazar.com/free-houdini-training/

thank you so much.

[odiwxe]

at the risk of oversimplifcation...

 

1- what's a shader?

 

a shader is an operation performed at render-time (typically for the current ray being rendered, but not always).  in houdini, there are a number of possible shaders: surface, displacement, lights, shadows, and others i can't remember off the top of my head.

 

2- what's a material?

 

in houdini, a material is a essentially a container for connecting a surface shader and a displacement shader for convenience purposes.

 

3- what's a texture?

 

in cg speak, a texture is an image (usually) that shaders use to assist in the shading process.  the most common use of textures is to project them onto your geometry and apply them as a color map.

 

4- what's VEX and VOPs?

 

vex is the custom language that shaders are written in.  vops is a graphical interface to assist in the creation of vex code for artists.

1 - great,i thought shader is the material itself,i thought it's something i can SEE but now i see that it's just some calculations(a calculation for the surface,another for lights,another for shadows and etc) behind the scene,am i right?

 

2 - so the thing we call a Material in houdini is actually just a home for different shaders?for example there are different shader types(as you said for shadows,surfaces,light etc)in a material that we assign to an object?so we mean there are different shader types INSIDE the material that forms the object look when we say for example a "clay material",right?so if it is we can have our shaders just here and there(not in a home called a material)?

 

==> another question, i'm seeing in tutorials they use different contexts in one context,for example SHOP in SOP,doesn't it matter where we place a node?

[odiwxe]

1- what's a shader?

 

​The process of assigning the colour to a pixel by the renderer is called "shading". To do shading, it uses a "shader". The shader has various input parameters that allows for artistic control over what the shader does (like diffuse colour, roughness, etc.). In Houdini, I think of the bundle of specific parameters along with the shader, as "material". That is, many materials can make use of the same shader, but with just different settings. 

 

it's interesting,so as fathom and you said a SHADER is a process or calculations(behind the scene?)that uses different INPUTS(like diffuse color,roughness,beeing reflective,refractice,glossy etc) that the renderer uses to render a scene, right? AND at the end all of these processes and calculations results in simply the "color" of each pixel in the rendered image? is it some thing like  different inputs > different shaders = materials ? for example for a concrete material we mean there are different shaders(surface,light,shadow,etc) INSIDE this material that each one having it's own inputs?

[old school]

I like to look at things in reverse from the render engine back to the scene file:

 

Houdini   \ IFD file                    \   Mantra

          / + file archives (abc, bgeo) /

 

What does a given Render engine take as input?

• It needs geometry.
• It needs a material bound to that geometry to describe how to render that geometry. How that geometry responds to light. What colour to shade the geometry with.
• It needs lights in the scene.
• It needs a camera. Everything is relative to the camera. All your geometry is transformed in to the space of the camera. Camera rules everything.

 

Everything else is how you set things up in Houdini, what associations you create between objects and lights, materials and objects, mantra properties.

 

Yes you can share a single shader between multiple geometry. You can apply per object material overrides of any parameter of any assigned shader. This allows you to change say the diffuse colour for each object assigned that base material.

 

You can control which lights illuminate what objects. You can determine what component of a light are used by an object: direct illumination, indirect illumination, reflections, refractions, shadow, etc. These "masks" are created in Houdini, a common term used to describe these set-ups.

 

 

Geometry type shaders generally contain four different components: surface description, geometry displacement, render properties, geometry procedurals. You absolutely need a surface description. The other three are optional.

 

Surface shaders are evaluated every time a ray hits geometry. The shader is executed, with all it's lighting function models defined (B*DFs such as diffuse, reflection, refraction, sss, etc). Each of these models can take input parameters that are expanded upon in the surface shader. This is the shader interface that the artist can modify to determine the look of that surface.

 

Mantra Pipeline

Back to Mantra now. When you launch a Mantra render, Houdini generates the ifd file, Mantra loads in the ifd file. It finds the camera.

 

Everything is now in camera space.

 

Mantra fires a ray from 0,0 or if you are using ipr, 0.5,0.5 and waits for a click.

 

Mantra then uses the bucket size to determine what geometry is being "covered" in that bucket looking through the camera. It traverses the ifd looking for geometry. If it finds geometry described in that bucket (bounding boxes usually defined in the ifd), it loads that geometry in, refining it if it has to (literally cuts the geometry up and sews things together propagating all geometry attributes, pretty cool). It then runs the displacement shader if there is one to refine the geometry further, down to the micro-poygon (pixel?) level, again sewing everything together and propagating all geometry attributes (geometry with sub-d property will also be refined to the micro-polygon level). If this is PBR, and you have wide open reflection masks and/or indirect lighting applied, all the geometry in the scene is loaded in as a ray can traverse well outside the current bucket.

 

Once all the geometry that Mantra has requested has been refined, the shaders are then applied. First the opacity component of the shader is run and all occluded geometry is tossed out that may not be directly or indirectly seen, then the surface shader is run on the remaining geometry. At all levels, any mantra property overrides are applied as defined in the ifd file as set up in Houdini.

 

IFD file

Most every render engine takes an intermediate file description. With Mantra this is the .ifd file.

 

For most lighting tasks, you can safely ignore the ifd generation as it is automatic. If you are interested in diving in to rendering, the ability to read these ifd files are important. Mantra doesn't read a .hip file. It reads .ifd files. Houdini generates the ifd file. Mantra reads the ifd file.

 

As I mentioned above, render engines need the geometry in camera space. The ifd file will add the proper camera transforms on the geometry written in to the ifd file such that everything is in camera space. The ifd file will also contain all the shader descriptions by name. If an object has a shader render property pointing to a specific named shader in the ifd file, Mantra will assign that material to that geometry.

 

How you set this up in Houdini determines how the ifd file is written.

 

Shaders

Shaders can be seen as render properties and surface-light descriptions. Geometry takes material shaders.

Lights also have shaders that determine how those lights are to be defined: area, point, IES profiles, light intensity based on direction, etc.

Cameras also have shaders. You can define the lens shape among other things.

 

There are some historic REYES scan-line shaders available as well. Historic? PBR's only been around for 6 years... LOL! Yeah some historic shaders such as atmospheric shaders, ambient shaders, etc. Safely ignore these SHOP types. There's always internal arguments how to diminish these nodes in Houdini as their use diminishes rapidly.

 

I still remember not 3 years ago Side Effects taking all kinds of heat from our user base for expending efforts in to PBR. LOL!!!! Nice to see that even PIXAR has seen the light and all render engines of merit have moved to PBR. Beware of google searches that are older than 1 year when researching lighting, rendering and compositing. There are a lot of archaic lighting set-ups that really aren't that relevant any more still kicking around. There may still be some schools with a half-implemented curriculum where they use PBR rendering but archaic scene setups and archaic compositing approaches that really don't fit with PBR style rendering. Always question everything relative to lighting these days. It is changing rapidly.

 

In Houdini, if you put down an object, light or camera, each of these has a specific parameter where you can assign a SHOP type shader. You need to add a companion SHOP in any /shop network in Houdini.

 

For almost all the work you do, just assigning surface shaders to geometry and leaning on the pre-defined automatically assigned shaders for lights and cameras is what you do. Nothing more. It's really easy these days.

 

Procedural Shaders

Procedurals in SHOPs... OK.

Procedurals are small programmes that Mantra can run on the geometry at the front end of the geometry creation stage. Procedurals are used to render alembic geometry, packed geometry references on disk, instance procedurals, etc. Procedurals extend the front-end of Mantra during the geometry creation stage.

 

Procedurals are also defined as SHOP type shaders. They can also be defined right inside a material as well. It's up to the discretion of the studio's lighting pipeline. The default Mantra Surface Material does not contain any procedurals at this time. Add Procedural SHOPs then assign to the object using the Properties parameter as a node path.

 

----

 

This is really skimming the thin surface of this critical part of the pipeline. As a lighter, this is already too much info as Houdini insulates the artist a great deal as to what is happening when they press Render. It's so simple these days:

• Add a camera
• Add a light or two. For Image Based Lighting (IBL), add a High Dynamic Range (HDR) image containing hemispherical environment lighting info to an environment Light.
• Add your geometry and add/assign the geometry shaders, tweak the shaders developing the geometry look (look development)
• Add a Mantra ROP, decide which render engine to use (PBR is highly recommended).
• Fire up a Mantra render in the IPR pane
• Render, tweak lights, render, repeat
• Render final sequence and don't forget to add any extra image planes in the ROP required by Compositing.
• Done

[old school]

As for VEX, it is a simple c-like language used to compose shaders. The shaders are "compiled" into an optimized format for use by Mantra. 

 

You can compose shaders using VOP nodes and this network is compiled as the ifd file is generated, or you can pre-compile your shaders. Houdini currently ships with shaders defined with VOPs and are not pre-compiled.

 

I personally pre-compile all my VOP shaders for efficiency.

[danw]

As for VEX, it is a simple c-like language used to compose shaders. The shaders are "compiled" into an optimized format for use by Mantra. 

 

You can compose shaders using VOP nodes and this network is compiled as the ifd file is generated, or you can pre-compile your shaders. Houdini currently ships with shaders defined with VOPs and are not pre-compiled.

 

I personally pre-compile all my VOP shaders for efficiency.

 

Hey Jeff,

I've wondered about this before - I understood it as VOPs automatic render-time compiling meant it didn't impact render times in any significant way.  Have I gotten the wrong idea on this?  Do un-pre-compiled shaders have a major render-time cost?

 

...and if so, what is involved in compiling them?  Is it just a case of using the "Compile VEX Code to Shop Type..." option?

Edited August 22, 2014 by danw

[old school]

VEX Compilation affects ifd generation times for every frame you render in the sequence. The compiled shader is embedded in the ifd file.  If you have a lot of complicated shaders, then yes this will impact overall render times as ifd generation is a part of the process. This is wasted cycles imho as you are re-compiling the same shader over and over and over again. Add up the seconds per week and then apply that to the electricity required to fuel and cool your machine/render farm, it adds up to a pretty penny. Pre-compilation of common shaders is a must imho.

 

 

The default Mantra Surface shader available in the material pane is not pre-compiled. If you have a lot of Mantra Surface shaders in your scene, each with different settings, each one is it's own network therefore as far as Houdini is concerned, unique. Each one of those Mantra Surface shaders must be compiled and injected in to the ifd file.

 

If you create your own Houdini Digital Asset (HDA) from the Mantra Surface shader, then things get a lot more efficient. Using multiple copies of that pre-compiled shader in your scene to apply different materials to your objects offers a lot of benefits. First all your materials are defined by a single asset. There is only a single material definition saved in the ifd and since this is pre-compiled, is simply added to the ifd file. All material tweaks are now injected in to the ifd per object as parameter overrides on your pre-compiled shader. When you move to Packed Primitive workflows, using a small subset of pre-compiled and managed shaders makes managing renders oh so simple. The ifd refers to the shader definition on disk, fetches the shader code, done.

 

The only downfall is if you wish to edit the Material asset, all the assets will receive that change, if that is a downside. The work-around is pretty simple: just create a new asset from the original material and edit that. Allow editing of contents on the asset, dive in and edit the VOP network. When you Match Current Definition, the asset will compile at that point. There is also the versioning of assets you can employ mantra_surface_jw::001 vs. mantra_surface_jw::002. This is why versioned assets were created so you can reference shaders with the same "name" yet point to different shader asset definitions on disk. This keeps backward compatibility intact. If you want to know more about versioning of assets, let me know.

 

 

You can take any SHOP that is not a pre-compiled asset and turn it in to an HDA, no different than any other asset.

 

Steps to convert the Mantra Surface shader in to an asset:

1. Add a Mantra Surface shader to /SHOP. Go to that network location.
2. RMB on the top SHOP and choose the menu option to Create Digital Asset...
3. When prompted, give the shader a unique name and label. In case of Mantra Surface, you can give a name mantra_surface_dw and the name Mantra Surface DW and save it to disk with the name mantra_surface_dw_001.otl (for my sand-box work I always version otls by names on disk). Make sure to save that material somewhere in your HOUDINI_PATH. $HOME/houdini13.0/otls works well for home use. If this is in a small facility and $HSITE is valid, why not put it in $HSITE/otls if you have permissions. Press Apply
4. You will get a warning dialog as to what to do with the parameters. You can safely choose "Destroy All Spare Parameters" if you wish even though it sounds harsh. What the warning is trying to tell you is that when you create an asset directly on top of a material SHOP, the parameters defined at the top SHOP level are intrinsically tied to all internal Parameter VOPs. Changes to any Parameter VOP inside the VOP network will result in immediate changes to the top level SHOP. The opposite is also true, making a parameter change in the Edit Operator Type Properties will actually change the internal settings of the Parameter VOP responsible for setting this parameter. That is all. Again "Destroy All Spare Parameters" is fine. This dialog is there to protect any custom spare parameters you may have added to the Shader as you were developing it. Say you added a custom spare parameter that drives both diff_clr and refl_clr, you'd loose that if you chose the "Destroy..." option but would be kept in place if you chose the first keep parameters option.
5. Now in the Edit Operator Type properties, in the first main tab, right at the very bottom is an option innocently called "Save Cached Code". Enabling this option will pre-compile the code every time you press Apply or Accept. Do that.
6. Lock up the asset by pressing Accept in the Edit Operator Type Properties Dialog.

Now you are working with a pre-compiled Mantra Surface and can reap all the benefits of working with pre-compiled shader assets.

 

Diving in to a pre-compiled locked off shader gives you a nice yellow sticky note telling you that the shader is pre-compiled and to unlock to edit, which you can do if you wish.

 

Duplicating this pre-compiled locked Mantra Surface shader will give you many benefits. Which begs the question: why doesn't Side Effects ship pre-compiled Mantra Surface?  I presume it is to offer the artist no barrier to edit the surface shader. Back in the days prior to PBR, yes this was common. These days, not so much at all. If you want this, submit the RFE request to support and add your voice to the list.

 

 

As an aside, I would like to see this compiling of shaders to HDA's streamlined an awful lot. To a single menu item and a single pop-up to make it real easy to sew up a material in to a material asset. The benefits are too all encompassing for us not to do this imho.

[old school]

There is one warning that I have to mention and it is a big one.

 

Every time there is an update to the Mantra Surface shader, one that is what I call a loose edit where VOP nodes are added/deleted within the network, your pre-compiled shader will NOT inherit that change. 

 

How often do we change the basic logic inside the Mantra Surface shader? I wish I could answer that by saying every major release but alas that is not the case. We have on occasion made subtle changes to the internal wiring of the Mantra Surface shader within a release cycle. This is quite rare and one would argue not announced loudly enough, if at all.

 

I would expect changes to occur to Mantra Surface across major builds, requiring you to create material packs for each major release moving forward.

 

I also see this hopefully as a short term issue.

[odiwxe]

 

I like to look at things in reverse from the render engine back to the scene file:

 

Houdini   \ IFD file                    \   Mantra

          / + file archives (abc, bgeo) /

 

What does a given Render engine take as input?

• It needs geometry.
• It needs a material bound to that geometry to describe how to render that geometry. How that geometry responds to light. What colour to shade the geometry with.
• It needs lights in the scene.
• It needs a camera. Everything is relative to the camera. All your geometry is transformed in to the space of the camera. Camera rules everything.

........

 

This is really skimming the thin surface of this critical part of the pipeline. As a lighter, this is already too much info as Houdini insulates the artist a great deal as to what is happening when they press Render. It's so simple these days:

• Add a camera
• Add a light or two. For Image Based Lighting (IBL), add a High Dynamic Range (HDR) image containing hemispherical environment lighting info to an environment Light.
• Add your geometry and add/assign the geometry shaders, tweak the shaders developing the geometry look (look development)
• Add a Mantra ROP, decide which render engine to use (PBR is highly recommended).
• Fire up a Mantra render in the IPR pane
• Render, tweak lights, render, repeat
• Render final sequence and don't forget to add any extra image planes in the ROP required by Compositing.
• Done

 

thank you old school,will you give me suggestion?where can i start learning houdini deeply?if the pipeline is modeling > animation > materials and textures > lights and cameras > effects > rendering and compositing then don't have any problem with modelling section,how can learn deeply the sections after animation?i want realism in my renders,i mean super realism,not interested so much in animation yet,i just to be able to render with production quality,how do i get there from scratch?

[fathom]

1 - great,i thought shader is the material itself,i thought it's something i can SEE but now i see that it's just some calculations(a calculation for the surface,another for lights,another for shadows and etc) behind the scene,am i right?

 

2 - so the thing we call a Material in houdini is actually just a home for different shaders?for example there are different shader types(as you said for shadows,surfaces,light etc)in a material that we assign to an object?so we mean there are different shader types INSIDE the material that forms the object look when we say for example a "clay material",right?so if it is we can have our shaders just here and there(not in a home called a material)?

 

==> another question, i'm seeing in tutorials they use different contexts in one context,for example SHOP in SOP,doesn't it matter where we place a node?

 

you can use a surface shader in place of a material when assigning materials.  materials can also include some properties that take precedence over object and scene settings (like displacement bounds, for example).  so maybe it would be best to think of materials as being collections of related shaders.  normally materials also promote specific parms of the underlying shaders and ensure they operate in concert (like for example, a brick material would push your brick sizes to both the surface shader and the displacement shader it would likely contain).

 

you can put shop networks inside of sop networks.  i tend to do this myself when creating custom shaders/materials.  i like that i can easily copy and paste the object containing the sops/shops without having to bounce around the ui.  but it's just as valid to put all your materials/shaders in the /shop level.

[fathom]

(double post)

Edited August 25, 2014 by fathom

[odiwxe]

you can use a surface shader in place of a material when assigning materials.  materials can also include some properties that take precedence over object and scene settings (like displacement bounds, for example).  so maybe it would be best to think of materials as being collections of related shaders.  normally materials also promote specific parms of the underlying shaders and ensure they operate in concert (like for example, a brick material would push your brick sizes to both the surface shader and the displacement shader it would likely contain).

 

you can put shop networks inside of sop networks.  i tend to do this myself when creating custom shaders/materials.  i like that i can easily copy and paste the object containing the sops/shops without having to bounce around the ui.  but it's just as valid to put all your materials/shaders in the /shop level.

thank you,you explain well,now i have a basic understanding of it, data is dataas ari danesh in sidefx tutorials told!