Image sequence projection for baking onto geometry

[markinglevfx]

Hi everyone,

I'm trying to solve a tricky problem in Houdini, and I could use some help! I need to project my live action plate onto some 3D dummy geometry and essentially bake down every individual frame. Essentially what I'm trying to produce is a 3D representation of my scene using the live action images as a diff map, because each projection should either overwrite or paint in new areas of the scene.

My geo matches the scene and the camera is tracked and solved from the plate and matches well. The shot doesn't include any moving objects so we don't have to worry about that.

I have experimented with the sop solver, the bake texture rop, and material layers, but I haven't quite made a breakthrough just yet - as these are concepts that I don't have very much experience with.

Any guidance or suggestions will be very welcome!

Many thanks

Edited May 18, 2016 by markingleukc

[Skybar]

You could just UV Texture SOP and use the perspective from camera. Then just apply your sequence as a texture and render?

[markinglevfx]

4 minutes ago, Skybar said:

You could just UV Texture SOP and use the perspective from camera. Then just apply your sequence as a texture and render?

I'll try to explain better, I think it's a tricky concept to visualise the first time around.

If you imagine: for every frame of my sequence as my shot camera moves through the scene, Houdini makes a new camera which is frozen in the main camera's position in 3D space. That new camera projects its corresponding frozen frame onto the geometry, as a new layer on top of the camera projections came before it. What you would end up with (in theory) is a 3D scene with colour information applied onto the matched 3D geo.

I'm sure there must be a solution using either the SOP solver or even using hscript, as you can easily iterate actions over each frame with these.

[Skybar]

Ah right I see, yes you could do that in a SOP solver. Like I described, but instead apply the texture to your point color and just keep adding for every frame. It would require your geometry to be fairly highres though to not be a blobby color mess.

[anim]

what did you use to track your shot? Syntheyes does have the feature that does this already so it may be easier than setting it up in Houdini from scratch

https://www.ssontech.com/tutes/tutetexture.html

 

but other than that yes, you can create point on the geometry for each pixel on the desired resolution in it's uvspace, then project color from camera on every frame and combine in Solver using your favorite math. Then save each point's value to corresponding pixel in COPs

Edited May 18, 2016 by anim

[Guest tar]

1 hour ago, markingleukc said:

I'll try to explain better, I think it's a tricky concept to visualise the first time around

It's a compositing 101 technique for removing camera motion, and, as Anim explained SynthEyes has the feature or we do it in Nuke every day.