Building a Pipeline

[LaidlawFX]

Hello,

So I first off hate building pipelines, but my curiosity is getting to the best of me as I wrangle one. Does anybody know of any books or resources on building CG pipelines? The more diverse information the better, a.k.a. competing styles. For instance; other software centric pipelines(maya as opposed to Houdini), different scripting language based pipelines(perl vs C vs Python), competing studios. My google fung shui is pretty badly off.

Thanks

-Ben

[Marc]

Hey Ben

Now there's a question with a million different answers . I've recently been through this and to be honest there is a lot of junk out there with regard to this.

These are some pretty good links:

http://cgsupervisor.blogspot.com/

http://raysinblue.blogspot.com/2010/12/vfx-pipeline-notes.html

The first one is very wordy, but the information is pretty good if you can get through it .

M

[LaidlawFX]

Hey Ben

Now there's a question with a million different answers . I've recently been through this and to be honest there is a lot of junk out there with regard to this.

These are some pretty good links:

http://cgsupervisor.blogspot.com/

http://raysinblue.blogspot.com/2010/12/vfx-pipeline-notes.html

The first one is very wordy, but the information is pretty good if you can get through it .

M

Thanks Marc.

This definitely opens some windows for me to jump out of in a hurry My fingers were probably instinctually saving me from this...

If there is more links to this stuff I would appreciate it... (my fingers don't want to type this last line)

-Ben

[Marc]

Those are the only ones I bookmarked, which tells you my estimation of the other ones I found .

Personally if I were starting a new pipeline today it would involve a lot of research into: Alembic, Katana and Python. I guess it all depends on what kind of pipeline you're trying to build though, that would change what approach you take somewhat.

What are you building? Perhaps we can help guide you a little.

Here's a nice doc from the Foundry:

http://www.thefoundr...na-white-paper/

M

[LaidlawFX]

I don't plan on carrying out too much more work down this road in the future, since it's not really my cup of tea. I need a couple more years before I would willing do this type of work as a primary, but for the sake of curiosity and as a case study here is kind of where I've been/at.

Small size studio, that plans on growing.

Houdini centric pipeline, Maya as the primary secondary package especially for outside vendors.

Obj since it's trustworthy, have gotten alembic wet yet.

Personally I've been mostly concerned with the asset management side of things as opposed to the broad stroke called pipeline.

We have enough "pipeline" in and off itself that it is beyond one persons scope.

I am not a coder, but I can figure out the problems that need to be navigated, and can acquired the code I need. But modern code like python is the easiest to digest, more ready examples, and its cross platform nature.

Financially small studio so we can't afford a package like katana nor the resources to put it into effect. The financial outlook for development work is kind of a one sided look based on the immediate needs versus investing long term at the cost of the present.

So native renders for cost, so luckily Mantra is really good for that.

And render ques like rush, alfred, and deadline.

Comp is Nuke/Shake, no flame or inferno suites.

The day to day work is a lot more fluid than big studios, since while tasks can be described as compartmentalized, the shots will role between artist more often, than in the step by process of a big facility.

Pipeline is managed and built based on the challenges needed. Stuff needs to be implemented and change as the project evolves.

One offs are common at first and then once a tool/asset is used multiple times it then gets rolled into a supported tool/asset.

Definitely different outlooks on how the pipeline should grow and how it is used, which has caused some good creations, but has left some stuff on the floor.

The artist using the system have a diverse experience basis and have certain expectations, since we're seemingly there, but not by any means fully.

Pipeline 1.0 is carrying us through production. 2.0 never got off the ground before the heavy part rolled in. Regrets and reality are common with that, since, you can only do soo much in certain time periods, but can always ask for more.

So that is kind of what is built/building. Definitely left it a little general as these were common points from where I've been, so it will help others if they dig this up.

[Andz]

I'll be following this thread closely. Thanks for the links Marc.

Laidlaw, what about editing, will it also be done in house?

[eetu]

I'm working on a new version of our pipeline here, too. Interesting stuff, but in the end every house needs to tailor stuff to fit their style/work/size/people.

Katana looks awesome, but I think we might be a bit too small for it, too :\

One link more, http://williework.blogspot.com/2011/06/pipeline-basics.html

[LaidlawFX]

I'll be following this thread closely. Thanks for the links Marc.

Laidlaw, what about editing, will it also be done in house?

Currently editing is done in house. FCP style for finished renders. Sound recording is out sourced, though the knowledge base is there just not enough hands on deck/time.

Editing on the side of pre-production needs to be strengthened though. Generally we need to do better storyboard/script editing up front, and get better lock down on client approval. Can't afford to do it the Pixar methodology, but you can't give control to a producer/financer/director that will redirect months of work, on a whim. For instance, since it is easier to edit stuff they think the work behind a roll cut and re-time is the same as it would be in 3D. The misunderstanding of rendering alone is crazy in that vein.

-Ben

[LaidlawFX]

In the vein of thinking, sharing, and feedback, I'm gona break down a couple sections of pipeline as I understand it. I haven't had the free time to learn from the links yet, just skimmed em, the stuff is pretty good though. I wish they offered a class on this stuff where I went to school(s).

Directory Structure

This is a pretty common structure mod-ed a bit to be more conversational. I'm gona travel down one branch of this structure.

Every day I need to cd into a directory like this; to get to my shot and setup my job environment( variables). Amazingly there is a lot of though put into this directory structure than I would have imagined. I found out more each time I see it being used differently. Plus it can be pretty flexible and tailorable from individual, small co, and big co.

cd /m/cc/job/scns/1001/anim/block/v001/ben/

Usually I have a-couple aliases setup to dive me in faster than typing out the whole thing.

A more descriptive option

/parentDrive/projectAcronym/HumanVsMachine/Breakup/Shot/Department/DepartmentTask/Version/Artist/

So there are bout 5 major sections, and 4 always tailored parts after that.

/m (Parent Drive)- This is the parent directory, and/or drive letter depending on OS. Usually 1 letter, symbolic of the company, or some random though. Like ddc for digital domain commercials, or s for Sony(haven't worked there). It usually avoids the lower alphabet like a, b, c. C since it is the default drive for windows based system. This allows for one level of aliasing and swap-ability above everything. Also can set company specific environment variables.

/cc (Project)- This is the projects directory. This is usually a very short acronym for the project. There are allways multiple projects going on, either in dev, current, or old. You could have multiple Demo Reel pieces in process, multiple commercial, multiple features, etc.

/job (Human Vs Machine)- This is a split up from human files and computer generated files, jobs and pixs, respectively. You could theoretically delete all of pixs at the end of the show and be able to hit render from the human files to rebuild pixs. Still in theory at most places, but with the right settup this is really useful. Job and pix is a parallel structure. The levels are quite often the same, but sometimes the use of that level of directory may be named something different. Most often pixs is where all your rendered images will, be this can often bloat to terabytes including images and ifds/ribs. So there a lot of scripts that will culls this info out, where as jobs you want to try and save your .hips, .ma, .nk, .shk, etc...

/scns (Breakup)- I have no clue what to appropriately call this level of the directory. I feel like it could be removed, but it has proven useful, so I call it in-affectionately the Breakup. This is where scenes, assets, scripts(bin), users, resources, transfers. Scenes contains each shot the project, assets contains models or effects that are not tools that are common to multiple shots with their outlying parts. User is the slop directory where people can experiment with stuff in their own manner, but still have the ability to use it as part of the regular directory structure. Resources are project common useful things that haven't been rolled into something in-particular. Transfer is a project specific area for erroneous transfers, some one doesn't know where this fits into "grand scheme" they drop it here after an e-mail or the person next to you says hey buddy.

/1001 (Shot)- This is the shot. I've scene and talked a lot about the different ways the shots are broken down and labeled or number, so there is no common thread. I would hope that a script locked down and you could break it down based on the script based on reels(20min time frames), or story marks or sequences. Usually they have a buffer between numbers for shots that get combined or split there is room. If you have ever done a script breakdown, and looked at the final script version with all it's color insert and banged up nature that is pretty much what happens at and below this level. Also below this level a software centric pipeline may take into effect like a maya studio may base the below structure on the default maya project directory (I personally do not like).

/anim (Department) - This is the department. At a big studio this is usually well defined, but at a small studio or an individual project this is where stuff becomes more fluid. It is hard to wear multiple hats and do thing in parallel paths when you fluidly put something together. So I would define this for a small studio as something that has contractually been met for approval. Layout, Anim, FX, Lighting, Comp, Final.

/block (Department Task) - This is the task of the department. For animation, it would be something like previz, block, anim, final.

/v001 (Version) - This might be combined with task of the department to limit folder hierarchy. This is known as an output event. If you show something to your supervisor at dailies, or something to a co-worker that looks good. You save that version and go to the next version. Encase several versions latter your supervisor says go back to the older version. This is often confusing if you like to save often since a program crashes, like maya. Houdini will create a version in backup every time you save the file. This is a lot cleaner, it goes against instincts of maya/many users, but you can always just save your extra OCD saving in the backup directory.

/ben (Artist) - I ain't a fan of this anymore, since it causes their to be an extra folder directory, and the constant potential to not work seamlessly with your co-workers by stomping on stuff and getting confused, especially when not working right next to each other.

This structure is best scene from a nodal diagram(not just because we are Houdini users), but it's hard to visualize it as a whole any other way. Need more time to show that.

The file names come next but are linked to the structure of where they come from. The pixs directory is heavily like this. And the asset directory is a solid parallel modification of this structure, and has a lot of nuances especially when tied up to otls, or references in maya.