Jump to content
ajz

Python's multiprocessing

Recommended Posts

Hello.

Has someone been able to successfully make use of Python's multiprocessing inside Houdini? Last time I tried, it began opening one Houdini instance per started process.

Share this post


Link to post
Share on other sites

I'm not sure I understand. That's exactly, what multiprocessing is supposed to do. Start a separate process to do the work.

Share this post


Link to post
Share on other sites

Hello Edward.

I expect it to run multiple Python processes within a single instance of Houdini. Not open multiple Houdini windows each running a single Python process.

Any attempt of utilizing Python's multiprocessing module inside Houdini ends up with the latter.

 

Unless I'm doing something terribly wrong, of course. ;)

Share this post


Link to post
Share on other sites

Are you talking about launching several threads from a single python script?

 

From my experience, Houdini has two types of python threads. The ones that run in the interface which appear to be locked. This means If you place your code in a button and click the button, you lock up the process until that button script completes. The other option is to run your code from the Python Source Editor window. This seems to be a non-locked thread. Thus you can launch a process and have it running in the background while still using the Houdini Application.

Edited by Atom

Share this post


Link to post
Share on other sites

I'm talking about creating several Python processes inside Houdini. Threads are a slightly different thing.

Let me explain.
I have a script that performs some operations on large ordered lists that store vertex numbers and attributes. I feed the lists with data acquired directly from hou.Geometry objects. Then I export the processed and formatted data outside of Houdini.
The problem is that lookup operations on ordered lists that contain thousands (to millions) of items are very slow, so I need to find a way to speed things up. Python's multiprocessing module (https://docs.python.org/2/library/multiprocessing.html) was the first and the most obvious choice to lower the lookup time because it allows to take full advantage of multiple cores, which would increase processing power at least a couple of times. I want to spread the calculations more or less evenly over several simultaneous processes.
However if I use this module inside Houdini, the application decides that it has to open its new instance. Then, in that new window it throws an error: "Load failed for F:/Users/Artur/1232. No such file or directory" (last number is random). It gets worse if I use multiprocessing pools, because then Houdini gets even more confused and opens even more of its windows.

I could probably rewrite the script to use dictionaries (hopefully) or push the collected data for processing outside of Houdini, but it would be more convenient (read: requiring less additional work...) to have the job done inside the application. Hence I am curious if someone was able to find a workaround and managed to get Python multiprocessing module up and running in H.

Share this post


Link to post
Share on other sites

You realize that the Python interpreter is *NOT* a separate process in Houdini, right? Since it's part of the same process as Houdini, the multiprocessing module can *only* fork the Houdini process. Since you need to export the data out of Houdini anyhow, then you should probably just do what you want outside of Houdini. If you're doing something inherently SIMD, Python is the wrong language to use anyhow. Try VEX.

  • Like 1

Share this post


Link to post
Share on other sites

You realize that the Python interpreter is *NOT* a separate process in Houdini, right? Since it's part of the same process as Houdini, the multiprocessing module can *only* fork the Houdini process.

Frankly, I didn't notice this, but it explains a lot, Edward.

I spawned a Hython subprocess (with hou module support) as Houdini's child process, from which I can now use multiprocessing. To communicate between child process and Houdini I use RPC.

After some initial testing it seems to be working fine, so I guess the problem is solved, unless something breaks.

Since you need to export the data out of Houdini anyhow, then you should probably just do what you want outside of Houdini. If you're doing something inherently SIMD, Python is the wrong language to use anyhow. Try VEX.

I'm still very fresh with VEX.

Share this post


Link to post
Share on other sites

After spending a couple of hours with Hython subprocesses and RPC I must admit that this wasn't a good idea. Processing data via rpyc turned out to be quite slow, up to such level that it would probably negate all benefits of fully utilizing multiple cores. I also quickly reached maximum chunk limit when fetching large lists from Houdini. Lastly, I noticed that all exception handles were ignored and led to script being aborted.

Not wanting to loose any more time looking for workarounds, I ended up optimizing the code and running it directly in Houdini's single process.

Share this post


Link to post
Share on other sites

I think I already experimented with threading module in Houdini some time ago, and if memory serves me right, it also opened multiple Houdini instances. I'll give it another try though, because you never know.

Share this post


Link to post
Share on other sites

Perhaps you might find luck with multithreading, as opposed to multiprocessing?

https://docs.python.org/2/library/threading.html#module-threading

 

I think it will end up the the same problem of global lock python interpreter. 

 

There are number of options to optimize your code even inside Houdini/Python (general idea: avoid calling or creating python objects in a loop), pre-processing in VEX, or using some numerical module as it's mostly probably written as C extension. Also: Cythoninlinecpp, hou.runVEX

 

You may attach some part of the problem here as a benchmark / challenge ;)

 

sie ma,

skk.

  • Like 1

Share this post


Link to post
Share on other sites

I think it will end up the the same problem of global lock python interpreter. 

 

There are number of options to optimize your code even inside Houdini/Python (general idea: avoid calling or creating python objects in a loop), pre-processing in VEX, or using some numerical module as it's mostly probably written as C extension. Also: Cythoninlinecpp, hou.runVEX

Cześć Szymonie.

Now I know what I will read while sipping my morning coffee. :)

 

You may attach some part of the problem here as a benchmark / challenge ;)

Well, the problem area will probably be trivial to you Houdini Experts, because it all boils down to a Wavefront OBJ exporter that supports vertex normals and materials. Houdini's default gwavefront executable does not export them. So I wrote my own. However, initially for the sake of simplicity and because at that time I was only a couple of days into Houdini, I decided to use lists instead of dictionaries. Lookup times of lists grew proportionally to their lengths which resulted in extreme export times for highpoly geometries.

After swapping lists with dictionaries, export times dropped like millions of times. It now takes approximately 25 seconds to export a 87.360 poly mesh, or around 5:25 to export 1,400,000 one. Much slower than gwavefront exporter (50 seconds, but without vertex normals and materials check), but not even remotely as slow as that unfortunate version with lists...

I chose Python, because that's one of the languages I feel very comfortable with. I realise that there are faster solutions though.

Share this post


Link to post
Share on other sites

VEX if applicable is inherently threaded so that would be the first thing I try. If you must use Python I'd do it outside of Houdini and use concurrent.futures in Python 3 if possible since it's so much more user and developer friendly.

 

https://docs.python.org/3.3/library/concurrent.futures.html

 

Not to mention all the command line argument improvements in Python 3 which make it that much easier to do stand alone scripts.

Share this post


Link to post
Share on other sites

Use:

 

import platform

import multiprocessing

if platform.system() == 'Windows':
    multiprocessing.set_executable('C:/Python27/pythonw.exe')
elif platform.system() == 'Linux':
    multiprocessing.set_executable('/usr/bin/python')

 

to use the system python. Beware that Houdini 13/14 is using Python 2.7.5 so you need to install the same version.

 

With this trick I was able to run system python with multiprocessing.

Edited by E. Ozgur Yilmaz

Share this post


Link to post
Share on other sites

Use:

 

import platform

import multiprocessing

if platform.system() == 'Windows':

    multiprocessing.set_executable('C:/Python27/pythonw.exe')

elif platform.system() == 'Linux':

    multiprocessing.set_executable('/usr/bin/python')

 

to use the system python. Beware that Houdini 13/14 is using Python 2.7.5 so you need to install the same version.

 

With this trick I was able to run system python with multiprocessing.

I'm not sure you will be able to use HOM in system interpreter, at least you need a licence for that..

Share this post


Link to post
Share on other sites

I'm not sure you will be able to use HOM in system interpreter, at least you need a licence for that..

 

The hou module can be imported into any Python, but yes, it uses a license. I think by default it will grab a hbatch license.

Share this post


Link to post
Share on other sites

I'm having issues with multiprocessing as well, on Linux, where my child processes don't seem to do any work when the script is run in Houdini.

 

set_executable() doesn't exist on Linux so I'm wondering if there is anything else I can do to try to fix it?

 

-Jon

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×