why mantra renderer is so so so slow?

[zarti]

It is a term used to describe a renderer that can generate a sequence of images in a consistent way, without flickering, without antialiasing issues, without crashing, handle large volumes of data and manage memory effectively. It should also be able to render 4K sized images (if enough Ram). It should be able to produce shadows that are free of errors (i.e. holes, flickering, or ghosting). It should also be flexible enough to meet the needs of the production.

It basically means a renderer with really good reliability, consistent results and flexibility.

It has nothing to do with the features of the renderer (i.e. global illumination). For example, there are cartoon render engines that are classified as production renderers, because they are reliable enough to be used on a large scale film.

When does a render engine become classified as a "production render". It never really does. It's either marketing hype by the software developer, or it's the reputation it has in the industry. Pixar's RenderMan is the golden standard for production renderer. No other render engine has produced more seconds of 3D animation for feature film (that I know of). So you can express this as an equation, given X number of hours rendering by number of error produced by RenderMan. This becomes the ratio that other render engines need to match or surpass, and that's going to be very hard.

thank you hopbin9 for the explanation ! i really appreciate it .

[symek]

I feel bad I had responded to this thread, after I actually looked at the cgsociety news about cycles render engine. This is pure trolling Swann.

[zarti]

Stalkerx777 i'm quoting you but this is not addressed specifically to you .

as you might know , im new here , but anyway .. ill shot =)

From my experience, users, which say that mantra is so so slow, actually, never had real production experience in mantra. I know it, several years ago, i thought the same.

id be very curious to know ; during those years , what contributed mostly in Owning the confidence you have today with mantra ?

your previous background ? special access to some secret wisdom source , or it was just simple time passing in its course while the random info you gathered during those years got slowly self-organized in your brain ?

..

Most of dissatisfied users, have tried to render chrome/glass balls, or simple models in white room, with a little, or, in most cases, without any knowledge of mantra. They get long render times and very noisy image. And then they go to forums..... The problem is, they can't(or don't want) to understand, that mantra - is production render, aiming towards vfx production.

i'm not one of those dissatisfied , but sentences like this do not help anyone .

not those who ask for more speed@mantra ( this turns them back from where they came from . and they came here for a reason at least )

not Hou-Makers also ( which obviously want to make their app the best out there, possibly with more and more users every day goes by ).

another note;

if 'they' ask for more speed@mantra , keep in mind that 'they' have timeframe references from other renderers and parameters they turn on/off .

oc , in those renderers exist some acceptable-compromises with the quality / speed ratio , but i believe that that Ratio doesn't come out of Nothing ; it $olve$ an equation to another huge range of 3D-app users out there . also some 3D-apps are adding features or partial contexts similar to those which Hou is famous for .

in this point of my arguing i arrived at the most Important Question ;

> Do really Hou-Makers see a great possibility in this huge range of potential users , in order to make them Abroad the target range beyond the vfx Holy Grail ??

i do not read threads entitled like " why modelling is so so ... " , or , " why animation is so so ... " , just about mantra ...

if proceduralism in modelling , animation , simulation are perfect and very attractive in Hou i think mantra should be the same . in terms of speed , oc .

personally i'm able find a solution for any problem in modelling or animating context . i know it is there and go hunt for it .

while with rendering things aren't so 'obvious' , hence the request addressed to Hou-Makers to make mantra more fast .

maybe it is the lack of docs and examples. i'm sure it is to a certain degree, but ...

Ability to render huge amount of geometry with dof,displacement and motion blur, and great flexibility over all - is the most valuable features.

...

absolutely true and that's why i'm here , Stalkerx777 .

i'll stick to Hou-Makers and wish to them the best while contributing even with posts like this .

..

and now just for the fun of it ; do you know why i haven't opened yet mine own thread of " why mantra ... ?!!!! " ?

simply because rendering with motion blur and dof turned on in Hou ( compared with my previous 3d-app ) is as 'easy' and enjoyable as crushing soap bubbles with a hammer !

.cheers

[dyei nightmare]

well, im agree with both sides on this topic, mantra is renderer like renderman where you can build your own shaders, where maybe you dont have a practical solution to render fast and pretty like vray, i do understand that big companies get a lot of beneficts from this kind of render engines...

but for a single artist who wants fast renders for small personal proyects, should exist a some kind of presset or default set-up for fast and practical renders...

i been getting pretty fast and good renders from mantra playing around with shading models in vex, but, not everyone has the time or the interest to play with deep level stuff... some artist just want an one button render solution for personal and small little projects... something like v-ray...

and im sure that sesi could implement a fast render pressets or assets for those people who want fast and pretty renders...

because not everyone is a shader builder expert.

Edited April 26, 2011 by dyei nightmare

[pclaes]

When I come to think of the better looking things I have rendered with mantra I actually realize that anything of higher standards has been rendered on the farm. When I am testing at home I will rarely render super heavy scenes... mainly because it is not just the rendertime, but also the strain it puts on my hard disks, memory, cpu... I broke my last laptop with a particle sim... so don't really want to go down that route any more. You kinda want to have dedicated hardware to handle that. If I were to set up a home office I would build a file server with a few render/batch nodes attached to it. With HQueue these days there is no need to lock up your houdini any more, and those extra cores in your machine can be processing in the background (say 10 out of 16 threads) - as long as you have enough memory in your machine(s).

If at home I go up to 60% of targetqualite by pulling tricks with the HDK or shader optimization/ render parameters, that means when I have access to a farm I can scale it up further and go all the way. I try to optimize, but if your rendertimes start to go over 1 hour per frame it is just not fun to do it at home any more.

So using houdini for non-professional work can be restrictive, because it's not just the software, but also the hardware.

I like mantra a lot. The main reason is because it is to a degree predictable if you know how a reyes renderer works. Also when it stops rendering it is often because it runs out of memory. There are debug passes that let you analyse your frames in regards to the time each pixel took to render. Even the hole rendering out pointclouds for caustics or bounce lighting is fairly transparent... I mean you can bring that pointcloud into the scene and reverse engineer how the scattering was performed by analysing the pattern... How many render engines do you know that allow you to have a look at that? I am learning renderman in parallel with mantra. The other reason why I like mantra so much is because it is a separate application from houdini and I like playing with the HDK to make geometry at rendertime. I even like obusing the free mantra licenses to perform batch operations .

I agree the help file could do with a lot more info and a lot more in depth info about the underlying algorithms. I don't think it would hurt if they were to include how a reyes renderer works, or how this or that sampling algorithm works. Sure you have siggraph papers and books on the subjects, but why not include it in the docs online... Perhaps work in a "downloadable content" fashion. Where people that want the extended docs with all the graphs and maths can download this into their houdini build and have access to a lot more without bloating the daily builds. The cool thing is that a lot of these concepts could actually be implemented as example files in different areas of houdini. Some houdini users have built their own raytracers/raymarchers in a vopsop, or Mario's chromatic aberration masterclass, or evenly distributed scatter patterns in sops... or visualizing a kdtree or clipping against the camera frustrum, or pcwriting the points from the micropolygons... there are so many tricks and algorithms a renderer performs, and so many of those are geometry operations. I've said this before, but to be good at houdini is to understand 3d. If sesi wants to help their users, they needs to teach them about the principles and algorithms of 3d.

But yes, Octane like gpu accelerated rendering would be awesome! But how much can you actually load into the graphics memory of your graphics card... I've got a geforce 580 gtx at home (a better card than what I use at work), but even that card only has 1.5Gigs of ram on it. You would maybe be able to render simple scenes with it very quickly and that would be fun. Then again, Octane costs 100 euros or something like that? And Blender is free... exporter and formats already exists and are feasible when your scene is not too big.

On a side note: I am fascinated by "cheap" ways (both in terms of finance as well as powerconsumption/efficiency) of increasing computational power like this, but on the other hand I don't want to be a system administrator and build all that:

http://arstechnica.com/science/news/2011/03/high-performance-computing-on-gamer-pcs-part-1-hardware.ars

http://arstechnica.com/science/news/2011/04/high-performance-computing-on-gamer-pcs-part-2-the-software-choices.ars

And about the multi-core... they are probably working on that as a priority and I'm curious to see what 12 will bring. But even then, if you have access to a farm your perspective changes. A batch license for instance is linked to a machine, not to the amount of threads it has. So you build a dual 12 core machine with 48 gigs of ram and tell it to accept up to 6 processes. Each process is allowed two threads and a % of the ram. Now your machine is full-on using at least half its' cores in the worst case scenario and full load in the best case. But as mentioned above, that requires to have access to a farm (or run your stuff in the background) and to have a process that can be split into tiny chunks.

But different processes means the memory is not shared like in threads, so the IO will be a lot more intense so you need a fileserver and a network that can handle that. I just don't really know if houdini is an application for home usage (the price alone will scare most people away). It's good for learning at home and doing maths/tools/fx setups, but really rendering entire short films with it, doubtful. To me houdini users are power users. They need a lot of knowledge and a great deal of intelligence to apply that knowledge and then they need powerful hardware to support all the data and calculations they produce. At home you can train your brain, not so much your hardware .

[peliosis]

Houdini certainly is production oriented, but some speed eventually won't do much harm to that production .

I may be wrong but in last few years sesi push houdini towards single homegrown users and little shops.

Now these non Phd CG lovers not neccesarily are going to be lured by houdini world at the moment.

Many features demand so much patience and time that can be considered non existent when compared to other software.

No matter how powerful are CHOPs it's hard to prove it's unholy greatness to character animators when even a sample scene from great chops book moves at 4 fps on strong modern hardware.

Mantra power and flexibility could just be not a biggest deal for 90 % of normal 10 - 20 people commercial shops.

No matter how powerful, it really is slow and hard to setup for most of the scenes.

There are artists who can afford spending days on learning how to setup sss not to freeze to death on overlapping objects or different scales or anything. But more artists would like it to work like vray, just hit render and move the light. And 99 % there is no need to look at .pc.

If architectural scenes with plenty of moving emerging objects, changing light, plenty of foliage, gi, motion blur and dof

can be considered a tiniest production...that's also not where mantra would shine.

The only way to efficiently use multibounce gi is PBR. There is literally no way to use PBR in that case.

DOPS are very powerful, but there are no examples that show the target little shop it's true strength.

Nothing that will make them turn their eyes from lagoa or ice or fume or ncloth or something else.

And they are slow. Flexible, but slow, and unable to do large scale simulations as efficiently as a number of other less flexible soft.

Work in houdini and look at task manager on your dual 6 core...

Now go to softimage and do the same.

I wondered if we can use houdini as our main app.

We are a tiny shop and we can't afford using houdini not because it is expensive (count the plugins and maintenance, 7k is really not that much), but because it is slow for what we and 90% of others alike do.

I know there are many people who put years in learning houdini for sheer LOVE of it.

It's not just a software but a brave hearted stubborn knight with his sword pointed at the black three chambered heart of you know who!

I'd really like it to be faster, no matter how production oriented it is.

[Korhon]

I find mantra very useful This is all mantra renders done at Gimpville.

We use it for everything these days, and it proven itself useful in all areas.

Pbr is a slow way to go, but it gives good result every time.

1hour to 5hours pr frame isnt a problem if you plan good and start rendering early.

http://www.youtube.com/user/Gimpville#p/u/6/FtNEs747VQg

http://www.youtube.com/user/Gimpville#p/u/4/SWyWxJ0WS6c

http://www.youtube.com/user/Gimpville#p/u/15/R2QKXNb2H0A

[hopbin9]

Korhon, awesome stuff.

[zarti]

yea.. , great staff , Korhon !.

now me OT : does there exist any other page with more links like those above ( a mantra compilation ) ?

[el_diablo]

I find mantra very useful This is all mantra renders done at Gimpville.

We use it for everything these days, and it proven itself useful in all areas.

Pbr is a slow way to go, but it gives good result every time.

1hour to 5hours pr frame isnt a problem if you plan good and start rendering early.

http://www.youtube.com/user/Gimpville#p/u/6/FtNEs747VQg

http://www.youtube.com/user/Gimpville#p/u/4/SWyWxJ0WS6c

http://www.youtube.com/user/Gimpville#p/u/15/R2QKXNb2H0A

Very beautifull work! However I wouldnt agree that 1-5 hours per frame isnt a problem. Its quite a problem for small shops with 1-10 employees, as the rendering resources are limited to maybe 50 processors or so (last few shops I worked for). And the turnaround time is quite quick, can be a day or less for a update to the client, maybe a week for a whole project. At most I would have 20-30 mins per frame and thats for a really complex stuff...

[hopbin9]

Very beautifull work! However I wouldnt agree that 1-5 hours per frame isnt a problem. Its quite a problem for small shops with 1-10 employees, as the rendering resources are limited to maybe 50 processors or so (last few shops I worked for). And the turnaround time is quite quick, can be a day or less for a update to the client, maybe a week for a whole project. At most I would have 20-30 mins per frame and thats for a really complex stuff...

The all common triangle "fast, cheap, quality". You can only pick two. That's true in any industry.

20 to 30 minutes per frame for all passes (assuming HD) is idealistic. I don't know of any render engine that can pull that off with good results on any modern processor. I think I average 1 hour or more per frame for HD with 6/8/12 core machines but my stuff looks good

If you have 50 processors and doing 20 minutes per frame, then you're pulling off almost 7 seconds of animation per hour. You can turn out an entire television commercial in one day. I'm sorry, but most shops with 10 employees aren't even close to being that organized to put something together that fast. I would assume the studios you've worked for are more disorganized then lacking rendering power. If people managed their time better, organized deliveries with the client better, then you'd have more then enough time to render everything.

I'd love to have 50 machines on my farm, cause then I could do 2 hours per frame

[pclaes]

20 to 30 minutes per frame for all passes (assuming HD) is idealistic. I don't know of any render engine that can pull that off with good results on any modern processor. I think I average 1 hour or more per frame for HD with 6/8/12 core machines but my stuff looks good

Maybe with Octane or Vray RT

He mentions:

It's rendertime for GTX 285. Now I've got two GTX480 and I think that render takes less then 10 min per frame - It's not so long for HD. I could render this scene in any render, but octane is different - I can't explain the magic. Usually I use Vray or Mantra in my everyday work, I'm using octane in my free time just for fun. Level of interactivity in octane really charms me. BTW if you want to use two cards in octane - you should turn off SLI

for this video:

I am tempted to try it out - for fun and personal projects at first because there are still a lot of production features missing -- and waiting for PBR takes a while even though it looks pretty. I guess for smaller projects with less amount of geometry/complexity this could be an option... but it's also still in Beta. Maybe when their rib support is there.

[hopbin9]

Maybe with Octane or Vray RT

http://vimeo.com/13902843

Oh yes, GPU rendering is going to change the game. I've seen GPU rendering stations with a dozen cards in them for sale, but it still seems like it's experimental right now.

I'm a fan of Jeff Patton's work. Most of his stuff is now done with IRay (http://jeffpatton.net)

Still, GPU render on a large scale is out of my reach.

[Andz]

I am tempted to try it out - for fun and personal projects at first because there are still a lot of production features missing -- and waiting for PBR takes a while even though it looks pretty. I guess for smaller projects with less amount of geometry/complexity this could be an option... but it's also still in Beta. Maybe when their rib support is there.

I have a license of Octane at home, it is in fact pretty fun to use it. But it still is a loooong way to go for production. Today I'd only use for still images and mostly product or architectural visualization.

[pclaes]

I'm a fan of Jeff Patton's work. Most of his stuff is now done with IRay (http://jeffpatton.net)

Thanks for sharing, I had not seen his work before. Some very interesting info and results indeed:

http://jeffpatton.net/2011/04/why-iray/

[Jason]

Hi,

We just recently rendered all of the interior factory sequence of Hop in Mantra, using PBR. This was probably around 200 shots, and we rendered much of it at 1.5k (not 2k because so many compositions needed some (2D) DOF anyway). We did not render the furry characters, only all the hard surfaces and liquids/smoke/etc.

We averaged about between 2-7 hours a frame - single threaded. We often rendered using 2-4 threads, depending on the machine's overall RAM config and so we ended up with frames 45min to 3 hours, for the most part.

This shows some of the factory sequences:

Our general settings were:

- 7x7 SuperSamples

- 1/9 Min/Max Ray Samples

- 0.01 or 0.008 Noise Limit

- 3 reflection and 2 diffuse bounces

- Gamma 2.2 colorspace

- Photon-mapping was used occasionally

I'll admit I'm not a fan of the design of the factory but PBR happily delivered what we needed. Some of the night scenes had upward of 400 light-sources in the scene. Of course there were some other things you need to do to squeeze Mantra for environments like this, but it worked very nicely - and hopefully it continues to improve.

[brianburke]

We just recently rendered all of the interior factory sequence of Hop in Mantra, using PBR. This was probably around 200 shots, and we rendered much of it at 1.5k (not 2k because so many compositions needed some (2D) DOF anyway). We did not render the furry characters, only all the hard surfaces and liquids/smoke/etc.

congrats, looks really nice.

super excited to see the boundaries of what pbr is capable of getting pushed to the next level.

[Korhon]

Great work Jason! I think the design of the facorty looks cool.

What did the ram peek for at those scenes we can see in the trailer? I find mantra very ram eating.

We got a 12gb ram limit at our houdini pool, and we do peek 12gb ram alot

Also what colorlimits do you guys use? I tend to put it to 2 for clamping out the worst noise. But it does kill the range a bit sometimes.

I cant wait to see what will happen with mantra/pbr in H12! If they speed it up a bit it will be the ultimate render.

[Macha]

We averaged about between 2-7 hours a frame - single threaded. We often rendered using 2-4 threads, depending on the machine's overall RAM config

Is there a reason why you render single threaded? Does that reduce memory?

It does look nice.

[Jason]

Is there a reason why you render single threaded? Does that reduce memory?

It does look nice.

Thanks; PBR really solved with a realistic touch which traditional lighting would've failed to produce so quickly.

As for single threading, here at R+H we never kid ourselves about how many CPU-hours we're using. We always speak in single-threaded times and, in fact, our queue viewer will always return the full total cpu time your job is taking (alongside the clock-time). This keeps you honest: for instance, a 20 cpu-hour render is a 20 cpu-hour render, regardless of whether you run it on a dual single-core proc, or a quad hexa-core. I've heard people say, "Yeah, that was only a 1 hr render.", but it took an hour on 12 procs which is, of course, a 12 hour render.

Texture access: Our scenes used a LOT of texture; some frames rendered easily upwards of 2000 x 2k maps. More threads = more mouths to feed = higher demand on network. Balancing the Mantra texture-cache (we typically set this to 2048) and number of threads, we found we got best performance from 2-5 threads, with 2 being the default.

The majority of our farm have 4 CPUs, and we found that the average RAM usage of a frame made it fit nicely on 2 threads and leave enough memory for another task or two on the other CPUs. This kept our queue pretty fluid and other shows and depts still got a fair share of render power without Mantra hogging up the place too much.

Our rule-of-thumb was 1 thread could take 4Gb RAM. With our 2Gb texture cache and heavy geometry we averaged about 6Gb, hence the default of 2. eg. If we had frames using, say 10gb, this was over 8 Gb of RAM and we'd increase the number of threads to 3 to accomodate. If the render machine had only a small amount of RAM available, it'd lock off the machine from new jobs and up the number of threads to the maximum.

As our request, SESI added this functionality:

Houdini 11.0.485: Add the ability to dynamically control the number of rendering threads in a tile callback. The new property "tile:threadlimit" (only available to tile callbacks) can be used to set or retrieve the limited number of threads being used to render. "tile:threadlimit" can only be set to values between 1 and "renderer:threadcount", and is reset between renders.

This was so that our queue manager software could dynamically change the number of threads down (and up again) as other tasks came and went during Mantra renders.

The tricky thing was, that we were rendering about 100 layers of deep raster information so that increasing the thread count would also increase memory usage -- sometimes by 100's of megabytes per thread. The only counter to this is to reduce the bucket size a bit (down to a min of ~5, less is too inefficient).

And so you know, much of the balancing was done on the fly by Yours Truly. We wrote some great PyFilter functionality to let me change the properties of rendering jobs on the fly without touching the hip file --- including making changes mid-frame due to our image-resume capability with our native file format! So very valuable I don't think we'd have finished the show without it!