Tracking, Plates and Live action General advice

[JodyLS27]

Hi there everyone, Hope you are well.

So I have been getting some advice from Industry Professionals about building a reel and I have been watching a lot of Houdini industry live streams and a they all say the same thing, CG with plates or tracking your shot.
Now I am very new to the FX side of things and I am progressing well with Houdini but I am not sure how things work on the Comp side other than the basic channel manipulation in Nuke and separating out your render layers in an exr file but anyways.
There are going to be a lot of questions coming up, I just want to really understand how things work.

So I know with tracking you place little markers on the ground (I.E a piece of paper with an X on it lets say) and then you can track that with a Software. Now what I want to know is what is the difference between Tracking and 3D tracking or is it the same thing ?

With Plates, is it just taking a picture of something, Cleaning it up in comp and comping in your CG element or is there tracking involved as well ?
I know there is things like lighting involved which you can match by eye I guess but is there a good way to do lighting ?

Another question is what are good softwares for tracking or Plates ?
I know there is after effects but I have also heard of Boujou and Syntheys or something.

Any advice would be really helpful.
Thank you in advance and have a great day.

Jody.

[StepbyStepVFX]

The Plate is the general name given to the footage you will work on, that can be « matchmoved » / « tracked » / « 3D Tracked », depending on what meaning you put behind, and on which you may composite 3D elements, or just clean, key, compositie 2D elements etc. with a proper compositing software.

Tracking : this can have different meanings, but they are linked. Usually, with a compositing software, you can « track » elements also called features (of the image) or points. This may be indeed some specific patterns printed and glued on set, but it can be other zones of contrast in your image, called « features ». Tracking just mean selecting a pattern in your image, and try to follow it from one frame to another. So its shape shouldn’t change too much over time to have a good track of your elements. The results of tracking is a bunch of points : 1 position for each frame, that describes the move of your pattern.

3D tracking or matchmoving : you can do « matchmove » on 2D elements (you translate a 2D element with the result of your track above, and therefore it « matches » the movement of the pattern you tracked - in 2D; and thus the name « matchmove »). But sometimes, you need to rebuild your scene in 3D : objects and movements of the camera. For that purpose, you will need to track (see above) a bunch of points on your plate. At least a bunch of good tracks, but more often a ton of them (many hundreds) if the movements of the camera are more complex. I won’t enter into details as there are tons of matchmoving tutorials (FXPHD especially, the ones on PFTrack for examples). You can track manually (you usually do it when there are some points / patterns / zones of contrast of interest in your scene, that you specifically want to get in your 3D space), and you can let the software find itself some of points to follow, but you usually needs to manually clean them (for example « false » points : like when two objects that does not have the same position in space appear to cross themsleves on screen, giving a lovely zone of contrast to follow for the computer, that « sees » that as a feature that does NOT exist in space...).

Once those 2D tracks obtained, you give as much info on the camera as possible, and then you let the computer « solve » the camera and the points : meaning, the computer will try to put those points in 3D space, try to put a virtual camera in this scene, recompute what this camera would see given its position & focal length & camera gate and the position of your points in space, and then compares this recomputed 2D positions of the points with the initial tracks you fed the software with. Then the computer iteratively try to adjust points and camera positions into space, to minimize their recomputed 2D positions with the initial tracks. That’s the way, after long computation times, and after a long 2D tracking process of « features », that you can have a cloud of points and a camera path, with estimated focal length and so on. From there, you can rebuild your scene (using your favorite modeling package), place nice 3D creatures / FX / objects and render them before compositing them on your initial / cleaned plate.

To matchmove, you can do it in NukeX (Nuke does not have the 3D features, NukeX does), which is also the reference for compositing. 

But for complex matchmoves, I like to use 3DEqualizer, or PFTrack. You also have Syntheyes and Boujou, that many people also use...

Of course, I have simplified : you may believe you have a good « solve », but you’ll see that what should be « straigth » or flat maybe bent etc., due to lens distortion. So you usually need to « undistort » your plate previously (using or not some reference pictures of grids taken with the same lens and camera that the ones used to film the footage). 

Concerning number of features to track in 2D to get a good 3D track : it depends on the length of your shot, the complexity of movements of your camera etc... The tracks should be as long as possible to cover many successive frames. You can have thousands of tracks that track features over 3 to 5 frames, before diying, but this may not be sufficient for complex camera moves... That’s why it must be thougth onset before filming :-)

I recommend you the courses on compositing of Eran Dinur on FXPHD (Nuke), but they have many, many good ones on Nuke as well, the courses on PFtrack (still FXPHD), you also have History of VFX by Matt Leonard (Fxphd again) and « VFX Foundations » of Tahl Niran (fxphd again, and this guys works at Weta), and finally this book to better understand the global VFX process :

The Filmmaker's Guide to Visual Effects: The Art and Techniques of VFX for Directors, Producers, Editors and Cinematographers

Hope that answers a bit your questions :-)

 

Edited October 10, 2018 by StepbyStepVFX

[JodyLS27]

@StepbyStepVFX thank you so much for your response and apologies for the late reply, This have just been a bit craze.
I will do a full reply tomorrow, I have to get my head around what you replied with. Its a bit of a brain twister