IFD Creation

[Anti-Distinctlyminty]

Hi all,
I'm trying to set up mantra with backburner to get some rendering done over our farm. That's the end goal anyway.
However, to do so I need to create IFDs for Mantra to render, and this process is a really significant bottleneck in the whole process because 1. The IFD creation is very slow and apparently single threaded, and 2. The IFD file size is quite significant.

While I can live with the file sizes, the real killer here is the workflow of having to have one machine create IFDs, which on my current shot takes approximately thirteen hours. Does anyone know why this extremely important step seems to be single threaded? To make this process feasible at all I had to use hbatch to manually open up ten copies of the scene (each eating RAM) and split up the IFD creation between them.

So, is there something I'm missing here? Are there some tools to make this process less painful? This is the first thing in Houdini that has frustrated me

Edited October 22, 2016 by Anti-Distinctlyminty

[goldleaf]

Yeah, just baking out IFDs can be slow and frustrating, I've been there before!

The key is to not store all the geometry in the IFD itself - that's where the slow-down and massive file sizes happen. When you use Packed Disk Primitives or Alembic files, the IFDs are light and speedy, as all the info doesn't get stored in the IFD; the IFD just references the files on-disk, and loads them when mantra renders the IFD. Does that make sense? For example, a 4.2Mb bgeo model when saved to the IFD creates a 4.3Mb IFD; but if you reference it as Packed Disk Primitive, the IFD is only 10Kb, and is nearly instantaneous to generate.

Check out the IFD Workflows page in the docs for some tips and examples; the hip file and geometry files ship with Houdini:

$HFS/houdini/help/files/ifd_workflows

Hope that helps, holler if you get stuck!

[Anti-Distinctlyminty]

52 minutes ago, goldleaf said:

...Does that make sense?

Oh yes, it does. I was supposed to write in my original post that I've looked into using packed disk primitives, but I don't think it's an option for my setup  The geometry in my scene is procedural and dynamic, so every frame has different geometry. I think my only hope was to somehow increase the speed at which IFDs can be created. 

Maybe I should submit an RFE for this to see if we can get the IFD creation to be dynamic. I seriously don't understand why it isn't considering how well designed everything else seems to be :/

[eetu]

38 minutes ago, Anti-Distinctlyminty said:

I don't think it's an option for my setup  The geometry in my scene is procedural and dynamic, so every frame has different geometry.

First write the geo sequence to disk and then refer to that as a disk primitive. That way the geo doesn't go through the slow python IFD generation bottleneck..

[symek]

29 minutes ago, Anti-Distinctlyminty said:

Oh yes, it does. I was supposed to write in my original post that I've looked into using packed disk primitives, but I don't think it's an option for my setup  The geometry in my scene is procedural and dynamic, so every frame has different geometry. I think my only hope was to somehow increase the speed at which IFDs can be created. 

Maybe I should submit an RFE for this to see if we can get the IFD creation to be dynamic. I seriously don't understand why it isn't considering how well designed everything else seems to be :/

 

IFD generation is as slow or as fast as Houdini cooking process for all nodes involved in a rendering. So any geometry which wasn't stored previously on disk has to be re-cooked on export, any shader which is not cached is going to be regenerated, which also takes some time - if you have hundreds of opened materials nodes in your scene. Thus:

- Any geometry which takes significant amount of time on cook should be pre-baked on disk (and ideally referenced via packed primitive/delayed load).

- Shaders should be closed and cached (when scene has lots of them) 

These are general rules one has to obey when doing heavy rendering on farm. This doesn't have anything to do with Houdini design. Any application which is asked to regenerate whole geometry before render, will do the same. The only difference is, that Houdini is the only application I know, which can generate gigabytes of geometry data from 1MB scene file, which you find so pleasing to work with. In any other application you first store data on disk, then render it from a file. Say, you have 1GB od geometry moving dynamically in 3dsmax sim, and you want to send it to Vray, you do what? (proxies?)

 

[Anti-Distinctlyminty]

18 minutes ago, eetu said:

First write the geo sequence to disk and then refer to that as a disk primitive. That way the geo doesn't go through the slow python IFD generation bottleneck..

I didn't write the geo out to files as I thought that was essentially what the IFD is already doing. Is writing geo out faster than writing IFDs?

[Anti-Distinctlyminty]

@symek Sorry, I misspoke in my original response. I said "Maybe I should submit an RFE for this to see if we can get the IFD creation to be dynamic. ", but I meant to say "multi-threaded", not "dynamic".

What I did in the end was to fire up about ten cmd windows all using hbatch to output IFDs for certain frame ranges. It definitely went faster, so maybe it's just something I have to create a script for perhaps?

Edited October 22, 2016 by Anti-Distinctlyminty

[goldleaf]

Yes it is faster than writing the geometry into the ifd. And you can 'parallelize' it by exporting different geometry/objects at the same time, or on different machines. Or if it's procedural, you can have each frame written to disk, on the farm.

[Anti-Distinctlyminty]

Ok that's good to know. I would set up the farm to create IFDs then render, but I only have a single Houdini license, so I can't run more than one instance of Houdini or hbatch AFAIK.

[goldleaf]

Your license shouldn't limit you from running multiple houdini/hbatch instances on the same machine. You could just fire off multiple 'Save in Background' from Geo rops, or manually run multple hbatch/hython sessions to export. If you run hbatch/hython manually, you could restrict each export to a certain number of threads, to not overwhelm your machine.

[symek]

What I meant is that practically speaking you've generated this geometry multiply times already, watching it in viewport, making previews from mplay etc. You will probably also render it multiply times, making tests, or layers, or fixing something. So  whenever this happens unless you store heavy assets on disk, you will have to wait for Houdini to generate it over and over again. It's horribly wasteful. Most applications don't allow you to do this. Whenever they generate heavy data (realflow for example), they force you to save it on disk on site, or they cache it inside a scene file, making it heavy and difficult to handle. Houdini is flexible here, what basically means it allows you to handle it in a wrong way.

The only case I an aware of where IFD generation is troublesome and it's hard in practice to pre-bake data are shots with multiply fur character. 

 

Multithreading of ifd generation would hit the same wall as multithreading of nodes' cooking in Houdini. Expressions anywhere and other context's interoperability makes is really hard to achieve (but pre-baking and packing geometry solve this issue to a great extend). 

[Anti-Distinctlyminty]

22 hours ago, goldleaf said:

Your license shouldn't limit you from running multiple houdini/hbatch instances on the same machine. You could just fire off multiple 'Save in Background' from Geo rops, or manually run multple hbatch/hython sessions to export. If you run hbatch/hython manually, you could restrict each export to a certain number of threads, to not overwhelm your machine.

Yup, that's what I ended up doing. Though this will rapidly become laborious, so I'll have to figure out a way to script hbatch to start multiple copies all outputting various frame ranges. 

[Anti-Distinctlyminty]

21 hours ago, symek said:

What I meant is that practically speaking you've generated this geometry multiply times already, watching it in viewport, making previews from mplay etc. You will probably also render it multiply times, making tests, or layers, or fixing something. So  whenever this happens unless you store heavy assets on disk, you will have to wait for Houdini to generate it over and over again. It's horribly wasteful. Most applications don't allow you to do this. Whenever they generate heavy data (realflow for example), they force you to save it on disk on site, or they cache it inside a scene file, making it heavy and difficult to handle. Houdini is flexible here, what basically means it allows you to handle it in a wrong way.

The only case I an aware of where IFD generation is troublesome and it's hard in practice to pre-bake data are shots with multiply fur character. 

 

Multithreading of ifd generation would hit the same wall as multithreading of nodes' cooking in Houdini. Expressions anywhere and other context's interoperability makes is really hard to achieve (but pre-baking and packing geometry solve this issue to a great extend). 

Yeah, I think my shots are just weird. The major workload is in the animation itself - think of hundreds of cells multiplying and dying, so the geometry is constantly changing, and the shading is relatively simple. 

My comment about the multi-threading was just that it seemed like there would be some sort of tool or process to do what I was doing manually (hbatch and render out an IFD range), but I can see that unless Houdini has the ability to internally clone an entire network, the threading issues are an unavoidable bottleneck. 

I'll see if using Goldleaf's suggestion of using 'Save in Background' in ROPs will help. I'm unfamiliar with this though, so I'll have to look into it.