Houdini Vs. Drqueue

[rdg]

How to integrate Houdini and Drqueue?

I added a wiki page (http://odforce.net/wiki/index.php/DrQueue) to archive snippets and stuff that might help.

Currently there are just a few commandline scripts, but I hope to contribute some more valuable information in the near future.

I didn't know how to upload a file to the wiki - so will post the shelf version of "toggle localhost" here

drqueue.shelf.txt

Controlling DrQueue from within Houdini using the python binding is pretty straight forward.

What is completely unclear to me is how a pipeline is set up.

Sure, there are some vague conceivabilities - but the whole picture is rather blurred.

Currently I set up a mantra node and attach it to a OTL - which should create the drqueue job definition and submit it to the manger.

There are so many ways to 'connect and integrate' nodes and OTLs - but not all are useful and valid.

What I am currently looking for is:

1: Setup a Mantra Node which can be used for "work/previews" - it renders single frames to ip

2: Get a Drqueue Node to use the overall settings of the "preview node" but set it to IFD and render a IFD sequence that outputs to image files.

...

Edited January 18, 2008 by rdg

[symek]

Thanks rdg! This is very helpful.

I wait for even more !

sy.

[rdg]

I spend the last two hours starring at $HFS/soho

Is it just me or is the documentation for SOHO rather non-existend -

[edward]

SOHO at a basic level is an python API for outputting to renderers. You can almost think of it as writing custom code for a ROP. It's meant to take information from Houdini and send it to the renderer. It shouldn't modify Houdini's state.

[rdg]

It shouldn't modify Houdini's state.

I almost finished - but I modify and restore some parameters of an existing mantra node.

I will post it here - maybe you can then tell me how to make this "more valid"

[rdg]

I see now how ugly my approach is - but I am lost.

nurse.tar.gz

The attached archive contains:

an OTL and the soho_command.

In the OTL I set some parameters:

a. which time range will be rendered

b. which Mantra Node should be submitted

c. where the ifds will be saved

d. how the output image will be named.

(e. some variables which theoretically help me to organise the projects)

The soho_command

1. checks if drqueue is running

2. if I specified a Mantra Node

3. if the Mantra Node is set to "Render Any Frame"

4. if the needed directories for ifd and images exist

6. modifies the Mantra Node to write IFDs and sets the output image

7. renders the mantra node

8. restores the original mantra settings

9. checks if it is the last frame iteration

10. if it is the last frame iteration it submits the job to drqueue

11. writes an useless protocol

I now understand that the soho_command is executed for every frame and a lot of steps are repeated over and over again.

===

Maybe there is a way to add an function to the regular MantraROP that submits a frame to drqueue if I check a checkbox?

Maybe just the same as an OTL without SOHO ??!

Edited January 18, 2008 by rdg

[edward]

Sorry, don't really have time to check it out. Just reading your setup, some things occur to me. What about having a mantra rop embedded *inside* your ROP HDA? Then you have much more control over what parameters are set by the user and how they're interpreted for output. If all you really need is to do some work before and after the IFD is output, then perhaps running a pre and post render script is enough, without the need for SOHO. Just thinking out loud.

[rdg]

Sorry, don't really have time to check it out. Just reading your setup, some things occur to me. What about having a mantra rop embedded *inside* your ROP HDA? Then you have much more control over what parameters are set by the user and how they're interpreted for output. If all you really need is to do some work before and after the IFD is output, then perhaps running a pre and post render script is enough, without the need for SOHO. Just thinking out loud.

thanks for your input. I guess you are right.

ROP HDAs is something I don't understand at the moment:

I guess they're really powerful IF you *know* what abstraction has to be established - eg you have a render setup/look that is used over and over again.

In my context the render settings change constantly.

So the pre/post render scripts combined with a HDA that sets the job/shot-variables and post-render submit the job to drqueue might be the way to go.

About SOHO:

When do I need SOHO?

If things are going to be to complicated to be solved by pre/post frame/render scripts?

[symek]

<br />When do I need SOHO?<br />If things are going to be to complicated to be solved by pre/post frame/render scripts?<br /><br />

As much as I understand SOHO is meant to suck out data from a scene, so basically you don't have to loop through geometries objects for example, reconstructing it triangle by triangle. The same with lights, cameras and so on. It gives you a snapshot of your scene at a current time with a normal Houdini speed in ROP context (you can't use it anywhere else - well, I tried and I couldn't).

Of course you don't have to use SOHO at all and use pure Python but this will force you to query every piece of data manually in its own loops - what makes this process slow and tedious.

Look on Honey thread here on odforce. At the very end Jason posted his early implementation of Houdini2Sunflow exporter. There is a nice snippet of code that shows how to deal with geometry using SOHO.

Finally, I don't see the reason of using SOHO to prepare DrQueue ROP. Since all you need are ifds, or hrender command, am I right

sy.

Edited January 19, 2008 by SYmek

[Jason]

Finally, I don't see the reason of using SOHO to prepare DrQueue ROP. Since all you need are ifds, or hrender command, am I right

You're right - and a little wrong if you're considering multi-host rendering. The best way I can explain it is by way of the Alfred ROP and Mantra multihost rendering. The current Alfred ROP has a pretty awkward limitation wrt this right now, IMO; I'll explain this later.

Alfred can assign multiple machines for the duration of a local process. In other words, it can reserve some machines and these hostnames can be fed as arguments to the -H argument of a Mantra process. When the mantra render terminates, the machines immediately re-enter the pool as available. This is an extremely useful productivity advantage - something that if you can use it, you should.

So in this context, it makes sense to be able to generate an IFD immediately from Houdini at the current frame and submit it to Mantra via Alfred/DrQueue. Shelling out to hscript/hrender/hython is unacceptably slow; on any decent production scene in a networked production environment this can take up to a minute - and thats not including any particle run-ups, etc. So for this, a SOHO implementation makes sense (or just plain HOM, really, since it doesn't doing any exporting of the scene) - which doesn't do the IFD generation itself but invokes another Mantra ROP to do it for you, and interacts with Alfred to manage the render of the frame.

Currently the Alfred ROP doesn't have this "immediate" mode to vanquish lag times in multihost rendering, but it definitely should! The whole point of multihost rendering is to get as many hosts as possible chewing on your frame as fast as possible.

So my point is really that there is a fairly good reason to have it as a ROP, and SOHO provides some convenient ways of accessing ROP parms quickly. It also provides flexibility to query the scene if you might need to for some reason. You could hopefully also, in the future, be able to trigger autoArchiving from SOHO, which is very beneficial to speed gains in multihost rendering. Sending huge models to every client Mantra process is a costly process and so if they can all load a (network-available) archive, things speed up immensely.

I have more tips on accelerating this too -if anyone is interested...

[edward]

In that case, what about having a SOHO ROP that basically does what IFD.py does plus some code at the end to submit it to the render farm? One idea is to directly customize IFD.py (and friends) to look for a custom render farm property that does this.

[Jason]

In that case, what about having a SOHO ROP that basically does what IFD.py does plus some code at the end to submit it to the render farm?

Well, Edward, this almost exactly what Alfred.py should directly support, yes!! It should support invoking the Mantra ROP directly (and this execute IFD.py) and then submit it to the farm - but it does not. Instead of invoking the ROP directly, it invokes hscript.

One idea is to directly customize IFD.py (and friends) to look for a custom render farm property that does this.

No, the last thing you want to do it modify IFD.py slightly. - you want to modify it significantly or not at all.. as a customer, you want to be synchronized with IFD.py bugfixes and improvements for as long as possible. As a member of SESI R&D you've been witness to the number of incremental improvements to the IFD.py pipeline as 9.0 as been improved and it's not an insignificant task to try to keep up with all these changes should we have custom modifications to the pipeline. Constantly updating a pipeline to stay synchronized is an arduous task - especially since SESI's changes to the pipeline are seldom fully documented. It's easy to think this way as a programmer I can understand but its really quite difficult to swalllow, as customer at the tail end of your whip.

That said, I'm totally appreciative of the power that SOHO provides... I've written 3 pipelines myself (Sunflow, Wren (R+H renderer) and most of Indigo (before my license expired )) in it that would not have been possible for me without it.

[rdg]

It should support invoking the Mantra ROP directly (and this execute IFD.py) and then submit it to the farm

that's what I tried with my nurse-soho.otl:

It evokes the .render() function of the assigned ROP and after the last frame is saved as IFD it submits the complete sequence to drqueue.

As I said - I don't know Alfred - so I can only guess that alfred can *group* frames in its interface. Something the drqman doesn't - that's why I prefer to submit frames as a range/sequence. Else I'd have my interface cluttered with single frame jobs.

No, the last thing you want to do it modify IFD.py slightly.

I agree - the cost of having the modified ifd.py synchronized is to high.

[edward]

Well, Edward, this almost exactly what Alfred.py should directly support, yes!! It should support invoking the Mantra ROP directly (and this execute IFD.py) and then submit it to the farm - but it does not. Instead of invoking the ROP directly, it invokes hscript. No, the last thing you want to do it modify IFD.py slightly. - you want to modify it significantly or not at all.. as a customer, you want to be synchronized with IFD.py bugfixes and improvements for as long as possible. As a member of SESI R&D you've been witness to the number of incremental improvements to the IFD.py pipeline as 9.0 as been improved and it's not an insignificant task to try to keep up with all these changes should we have custom modifications to the pipeline. Constantly updating a pipeline to stay synchronized is an arduous task - especially since SESI's changes to the pipeline are seldom fully documented. It's easy to think this way as a programmer I can understand but its really quite difficult to swalllow, as customer at the tail end of your whip.

First of all, this (rendering) is outside my area so you're not really talking to the right person here. I'm just giving my uninformed two cents. Perhaps another way is to have your own ROP that that calls IFD.py to do your thing and then submit it.

[rdg]

The whole point of multihost rendering is to get as many hosts as possible chewing on your frame as fast as possible.

why do kids always need to make their own mistakes to learn?

I now experienced what you were referring to ...

And the speed up of getting output from nodes while the workstation still tries to generate the ifds indeed balances a 'cluttered interface' with 6000 single frame jobs.

===

How do I get the "current frame" in a SOHO script?

I need to feed the current frame to the drqueue-command to render the right ifd.

But my SOHO script:

int(timeTo Frame(now)) is always one frame behind:

on frame 1 it generates foo.0001.ifd but sends foo..ifd to mantra

on frame 2 it generates foo.0002.ifd but sends foo.0001.ifd to mantra ...

any idea?

Georg

[rdg]

The frame mystery was based by rounding errors.

hou.hscriptExpression("padzero(4, " + str(hou.frame())

looks ugly, but gives me the value I need.

---

drqueue doesn't like my idea of submitting 6000 frames as 6000 jobs.

After job 100 it claims: No space in database.

Georg

[rdg]

I added a snippet to the wiki:

add or remove a computer to a pool.

drq_pool -a foo : this adds the local computer to the pool named foo

drq_pool -r foo : this removes the local computer from the pool named foo

[rdg]

I added a snippet to the wiki:

list the existing pools.

[rendertan]

Hi

I don't know if anyone's interested, but i remember reading a brief how-to at martian-labs website (it includes a couple of scripts and C sources).

For those interested, DrQueue for Mantra

[rdg]

Hi

I don't know if anyone's interested, but i remember reading a brief how-to at martian-labs website (it includes a couple of scripts and C sources).

For those interested, DrQueue for Mantra

Thanks!

I added a link it to the wiki page.