gRPC with Houdini HDK advice?

[Daniel_Daniel]

I found an interesting Java snippet, which generates a large three-dimensional array. I would like to generate different variations of this and send it to Houdini for further processing.

This 3D array barely fits into a single computer memory, so I would like to build a (bear with me) distributed system with a number of Java and Houdini nodes talking to each other via... gRPC.

I am not awfully familiar with gRPC, but it looks like an ideal candidate due to performance. I don't imagine that Houdini supports it, but how would one go about implementing it with HDK?

[symek]

On 12/17/2020 at 7:05 PM, Daniel_Daniel said:

I found an interesting Java snippet, which generates a large three-dimensional array. I would like to generate different variations of this and send it to Houdini for further processing.

Can't you just port Java snippet to VEX or Python/C++ and free yourself from the burden of inter-process communication? I know tinkering is fun, but can be very frustrating when involves real-time... anything.

If that's not possible 1) I would start from asking if those chunks single process will be occupied with have to communicate with each other? Do they have boundary condition set by their in-progress neighbors? Because if not, you can avoid distributed computing all together by slashing Java array into chunks and pipe it down to many Houdini's independent sessions (concurrent, not parallel, which is easier).

If that's not possible 2) I would consider trying Houdini's distributed simulation mechanics first. If you setup distributed FLIP, you'll notice that most tricky business is handled with two nodes called: Gas Net (Field) Slice Exchange, and afaik they will happily exchange any given point attribute or volume over the network's attached chunks. They don't ask what sort of computation is taking place, so you could distribute any VEX snippet or C++ operating on points/volumes over nodes (by just distributing something like a POP sim).

If that's not possible 3) I would look into Houdini Engine, which is built around  Apache Thrift.  In the easiest scenario you can run Python process (with hapi module) and connect it with any number of other python sessions. And those will have entire Houdini inside them running. If you insist on using Java as a data server, you could use Thrift on its side to communicate with Houdini Engine. Using Thrift just spares you time as it's already in Houdini.

If that's not possible 4) I would consider using one of Houdini's headers like UT_NetMessage or UT_NetStream for raw data transfer. You can even see the example of how the former one is used in VDB slice exchange.  In such case you would have to marshal  data for network transfer, there are number of options out there, including FlatBuffers, which is funny, because...

FlatBuffers has a builtin support for gRPC, so you get both at once, but gRPC, as name implies, was built for Remote Procedure Calls,  not transferring loads of data, so it might not operate as good as you wish with many gigabytes of it. Low level network in nice package (like mentioned HDK headers) may work better (or not)... Also something designed entirely to deal with terabytes over the wires like Arrow might be waiting for your call.

hope this helps,

skk.

ps frankly speaking if you would have been really badass, you would stick to OpenMPI

[Daniel_Daniel]

Wow, thanks a lot for taking a look at this. I'm nowhere near as experienced to know which of those options would work best yet... I'll just have to check all of them. Thanks for bringing up distributed simulations, Houdini Engine, FlatBuffers, Arrow and OpenMPI - I'll dive deeper into all of those. I agree that gRPC wasn't the right direction, there are plenty of better solutions for just moving plain data.

Just for reference here is the snippet I wanted to process with Houdini.

[symek]

Fractals on Processing! Very nice project, but definitely something that should be ported over to Houdini. The neatest version (imho) would use  OpenCL SOP. And distributing over network via DOPs is still applicable. Have fun! but hurry up, because some folks here might be interested as well

[Daniel_Daniel]

Thanks and I'm glad that you like it - I owe this codebase my architecture degree really. Full credit to an ingenious researcher Brent Werness (who I hope will write a license for this code at some point). It's a variation of cellular automaton and noise generating algorithms (I think it's a close "breed" to fractals).

I did try and got stuck porting this to VEX actually. Offloading this to multiple threads got in the way (I think) and simply copying this into Python didn't seem that beneficial. I would rather keep the original as it is and pipe only the end result (the 3D array) for processing in Houdini (setting up 'best practice'). After all, there must be tons of lesser-known gems similar to this on the Internet and I hoped to develop a more general solution first for dealing with them as they come and go. I realize that doing this directly in Houdini with OpenCL and DOPs distributed simulations would be easier though. 

Also if someone did port it to Houdini before me that would be actually pretty cool!

Edited December 19, 2020 by Daniel_Daniel

[Librarian]

This Looks Like This Or?
https://softologyblog.wordpress.com/2017/05/27/voxel-automata-terrain/


I seen This In OpenCL In Houdini  in Some Asia Forums 100% I have This Somewhere

Edited December 19, 2020 by Librarian

[Daniel_Daniel]

2 minutes ago, Librarian said:

This Looks Like This Or?
https://softologyblog.wordpress.com/2017/05/27/voxel-automata-terrain/

That's correct, softology did port his snippet soon after. Thanks for digging it out  I forgot the name of their website.

Edited December 19, 2020 by Daniel_Daniel

[Daniel_Daniel]

4 minutes ago, Librarian said:

I seen This In OpenCL In Houdini  in Some Asia Forums 100% I have This Somewhere

oh wow, yes if you could please deep search your browsing history to find this version that would amazing!

[Librarian]

@Daniel_Daniel have File somewhere in 2d and 3d
Here its some Link that you can Learn how to Adapt , Maybe it Helps ...gonna Find that File
https://zhuanlan.zhihu.com/p/54396666

[Daniel_Daniel]

1 hour ago, Librarian said:

@Daniel_Daniel have File somewhere in 2d and 3d
Here its some Link that you can Learn how to Adapt , Maybe it Helps ...gonna Find that File
https://zhuanlan.zhihu.com/p/54396666

excellent, thank you. So he goes straight up to OpenCL - that should give it an amazing speed up. If I translated it correctly he implemented the famous Game of Life, which should work fine on GPU, but the code I linked above uses also a bit more tricky diamond-square algorithm. I'm not quite sure yet how easy it will be to 'vectorize' it. 

Edited December 19, 2020 by Daniel_Daniel

[Librarian]

@Daniel_Daniel More Investigation
http://www.woutvanpoppel.nl/Project/OpenCLTerrain
http://luoxi.de/content/3d/11.html

[Daniel_Daniel]

1 hour ago, Librarian said:

@Daniel_Daniel More Investigation
http://www.woutvanpoppel.nl/Project/OpenCLTerrain
http://luoxi.de/content/3d/11.html

amazing, thank you! So there shouldn't be any reason for this algorithm not to work on OpenCL. That opens up an ocean of possibilities. I'm currently swamped though by another research, but I will definitely try looking into this as soon as possible.

In a more distant future I would like to integrate this type of iterative functions/fractals into architectural design. I think that there are a lot of opportunities for these gems in this sphere.

[Daniel_Daniel]

I am wondering how much overhead there would be in the serialization of such an array - Apache Arrow looks very promising:

 1. Importing csv files in the old GameDev's Python node used to take ages. I haven't tried the new csv node in PDGs, but the point is that there seemed to be a huge benefit in using bgeo. 

 2. Apache Arrow has Flight framework, but it only works with gRPC's protobuffs. As symek mentioned Houdini integrated with Thrift already, but it might be interesting to test if an inlinecpp/HDK module would yield any noticeable difference for a 6GB file for example. 

Conclusion: the world of (premature) optimizations knows no end.

Edited April 6, 2021 by Daniel_Daniel
website calls it a framework

[Daniel_Daniel]

from a recent Databricks presentation: youtube.com/watch?v=Jm9HC9y3MDA

 

Edited April 6, 2021 by Daniel_Daniel