Python's multiprocessing

[ajz]

Hello.

Has someone been able to successfully make use of Python's multiprocessing inside Houdini? Last time I tried, it began opening one Houdini instance per started process.

[edward]

I'm not sure I understand. That's exactly, what multiprocessing is supposed to do. Start a separate process to do the work.

[ajz]

Hello Edward.

I expect it to run multiple Python processes within a single instance of Houdini. Not open multiple Houdini windows each running a single Python process.

Any attempt of utilizing Python's multiprocessing module inside Houdini ends up with the latter.

 

Unless I'm doing something terribly wrong, of course.

[Atom]

Are you talking about launching several threads from a single python script?

 

From my experience, Houdini has two types of python threads. The ones that run in the interface which appear to be locked. This means If you place your code in a button and click the button, you lock up the process until that button script completes. The other option is to run your code from the Python Source Editor window. This seems to be a non-locked thread. Thus you can launch a process and have it running in the background while still using the Houdini Application.

Edited February 14, 2015 by Atom

[ajz]

I'm talking about creating several Python processes inside Houdini. Threads are a slightly different thing.

Let me explain.
I have a script that performs some operations on large ordered lists that store vertex numbers and attributes. I feed the lists with data acquired directly from hou.Geometry objects. Then I export the processed and formatted data outside of Houdini.
The problem is that lookup operations on ordered lists that contain thousands (to millions) of items are very slow, so I need to find a way to speed things up. Python's multiprocessing module (https://docs.python.org/2/library/multiprocessing.html) was the first and the most obvious choice to lower the lookup time because it allows to take full advantage of multiple cores, which would increase processing power at least a couple of times. I want to spread the calculations more or less evenly over several simultaneous processes.
However if I use this module inside Houdini, the application decides that it has to open its new instance. Then, in that new window it throws an error: "Load failed for F:/Users/Artur/1232. No such file or directory" (last number is random). It gets worse if I use multiprocessing pools, because then Houdini gets even more confused and opens even more of its windows.

I could probably rewrite the script to use dictionaries (hopefully) or push the collected data for processing outside of Houdini, but it would be more convenient (read: requiring less additional work...) to have the job done inside the application. Hence I am curious if someone was able to find a workaround and managed to get Python multiprocessing module up and running in H.

[edward]

You realize that the Python interpreter is *NOT* a separate process in Houdini, right? Since it's part of the same process as Houdini, the multiprocessing module can *only* fork the Houdini process. Since you need to export the data out of Houdini anyhow, then you should probably just do what you want outside of Houdini. If you're doing something inherently SIMD, Python is the wrong language to use anyhow. Try VEX.

[ajz]

You realize that the Python interpreter is *NOT* a separate process in Houdini, right? Since it's part of the same process as Houdini, the multiprocessing module can *only* fork the Houdini process.

Frankly, I didn't notice this, but it explains a lot, Edward.

I spawned a Hython subprocess (with hou module support) as Houdini's child process, from which I can now use multiprocessing. To communicate between child process and Houdini I use RPC.

After some initial testing it seems to be working fine, so I guess the problem is solved, unless something breaks.

Since you need to export the data out of Houdini anyhow, then you should probably just do what you want outside of Houdini. If you're doing something inherently SIMD, Python is the wrong language to use anyhow. Try VEX.

I'm still very fresh with VEX.

[ajz]

After spending a couple of hours with Hython subprocesses and RPC I must admit that this wasn't a good idea. Processing data via rpyc turned out to be quite slow, up to such level that it would probably negate all benefits of fully utilizing multiple cores. I also quickly reached maximum chunk limit when fetching large lists from Houdini. Lastly, I noticed that all exception handles were ignored and led to script being aborted.

Not wanting to loose any more time looking for workarounds, I ended up optimizing the code and running it directly in Houdini's single process.

[Jason]

Perhaps you might find luck with multithreading, as opposed to multiprocessing?

https://docs.python.org/2/library/threading.html#module-threading

[ajz]

I think I already experimented with threading module in Houdini some time ago, and if memory serves me right, it also opened multiple Houdini instances. I'll give it another try though, because you never know.

[symek]

Perhaps you might find luck with multithreading, as opposed to multiprocessing?

https://docs.python.org/2/library/threading.html#module-threading

 

I think it will end up the the same problem of global lock python interpreter. 

 

There are number of options to optimize your code even inside Houdini/Python (general idea: avoid calling or creating python objects in a loop), pre-processing in VEX, or using some numerical module as it's mostly probably written as C extension. Also: Cython, inlinecpp, hou.runVEX

 

You may attach some part of the problem here as a benchmark / challenge

 

sie ma,

skk.

[ajz]

I think it will end up the the same problem of global lock python interpreter. 

 

There are number of options to optimize your code even inside Houdini/Python (general idea: avoid calling or creating python objects in a loop), pre-processing in VEX, or using some numerical module as it's mostly probably written as C extension. Also: Cython, inlinecpp, hou.runVEX

Cześć Szymonie.

Now I know what I will read while sipping my morning coffee.

 

You may attach some part of the problem here as a benchmark / challenge

Well, the problem area will probably be trivial to you Houdini Experts, because it all boils down to a Wavefront OBJ exporter that supports vertex normals and materials. Houdini's default gwavefront executable does not export them. So I wrote my own. However, initially for the sake of simplicity and because at that time I was only a couple of days into Houdini, I decided to use lists instead of dictionaries. Lookup times of lists grew proportionally to their lengths which resulted in extreme export times for highpoly geometries.

After swapping lists with dictionaries, export times dropped like millions of times. It now takes approximately 25 seconds to export a 87.360 poly mesh, or around 5:25 to export 1,400,000 one. Much slower than gwavefront exporter (50 seconds, but without vertex normals and materials check), but not even remotely as slow as that unfortunate version with lists...

I chose Python, because that's one of the languages I feel very comfortable with. I realise that there are faster solutions though.

[Stalkerx777]

H14 vex can now work with files (read/write). http://www.sidefx.com/docs/houdini14.0/vex/functions/file_write

Also numpy (ships with houdini) can help you with huge arrays.

[lukeiamyourfather]

VEX if applicable is inherently threaded so that would be the first thing I try. If you must use Python I'd do it outside of Houdini and use concurrent.futures in Python 3 if possible since it's so much more user and developer friendly.

 

https://docs.python.org/3.3/library/concurrent.futures.html

 

Not to mention all the command line argument improvements in Python 3 which make it that much easier to do stand alone scripts.

[E. Ozgur Yilmaz]

Use:

 

import platform

import multiprocessing

if platform.system() == 'Windows':
    multiprocessing.set_executable('C:/Python27/pythonw.exe')
elif platform.system() == 'Linux':
    multiprocessing.set_executable('/usr/bin/python')

 

to use the system python. Beware that Houdini 13/14 is using Python 2.7.5 so you need to install the same version.

 

With this trick I was able to run system python with multiprocessing.

Edited May 21, 2015 by E. Ozgur Yilmaz

[Stalkerx777]

Use:

 

import platform

import multiprocessing

if platform.system() == 'Windows':

    multiprocessing.set_executable('C:/Python27/pythonw.exe')

elif platform.system() == 'Linux':

    multiprocessing.set_executable('/usr/bin/python')

 

to use the system python. Beware that Houdini 13/14 is using Python 2.7.5 so you need to install the same version.

 

With this trick I was able to run system python with multiprocessing.

I'm not sure you will be able to use HOM in system interpreter, at least you need a licence for that..

[lukeiamyourfather]

I'm not sure you will be able to use HOM in system interpreter, at least you need a licence for that..

 

The hou module can be imported into any Python, but yes, it uses a license. I think by default it will grab a hbatch license.

[jonp]

I'm having issues with multiprocessing as well, on Linux, where my child processes don't seem to do any work when the script is run in Houdini.

 

set_executable() doesn't exist on Linux so I'm wondering if there is anything else I can do to try to fix it?

 

-Jon

[Trancor]

I was running into problems with multiprocessing and queue in houdini/python in the past with the hou module.  Talked to Jeff at sidefx and said multithreading/queue the hou module can be unstable or all out not work.  Seems dependent on how the hou module is used in multithreading.  Since Houdini is using GIL for multithreading, you can try to tap into that. (Which I'm currently on the hunt to figure out, haha)
I've gotten multiprocessing and queue to work in shelf tools, just not using the hou module and addressing houdini nodes.

jonp, since you are on linux, you could always launch another terminal from houdini to run a command, like a batch houdini call.  When using the `terminal -e` flag, the terminal should self close when the command is completed.  Or at least does on my xfce os.
Sorry I can't be of more help here.

Edited August 3, 2018 by Trancor

[tricecold]

Not straight from inside Houdini and not for processing actual data multiprocessed but I got multiprocessing working inside hython quiet successfully. I am now trying to implement it as an HDA.

I would welcome any help

https://github.com/tricecold/hythonWedger

 

Edited November 26, 2018 by tricecold