Jump to content

Python's multiprocessing


ajz

Recommended Posts

Hello Edward.

I expect it to run multiple Python processes within a single instance of Houdini. Not open multiple Houdini windows each running a single Python process.

Any attempt of utilizing Python's multiprocessing module inside Houdini ends up with the latter.

 

Unless I'm doing something terribly wrong, of course. ;)

Link to comment
Share on other sites

Are you talking about launching several threads from a single python script?

 

From my experience, Houdini has two types of python threads. The ones that run in the interface which appear to be locked. This means If you place your code in a button and click the button, you lock up the process until that button script completes. The other option is to run your code from the Python Source Editor window. This seems to be a non-locked thread. Thus you can launch a process and have it running in the background while still using the Houdini Application.

Edited by Atom
Link to comment
Share on other sites

I'm talking about creating several Python processes inside Houdini. Threads are a slightly different thing.

Let me explain.
I have a script that performs some operations on large ordered lists that store vertex numbers and attributes. I feed the lists with data acquired directly from hou.Geometry objects. Then I export the processed and formatted data outside of Houdini.
The problem is that lookup operations on ordered lists that contain thousands (to millions) of items are very slow, so I need to find a way to speed things up. Python's multiprocessing module (https://docs.python.org/2/library/multiprocessing.html) was the first and the most obvious choice to lower the lookup time because it allows to take full advantage of multiple cores, which would increase processing power at least a couple of times. I want to spread the calculations more or less evenly over several simultaneous processes.
However if I use this module inside Houdini, the application decides that it has to open its new instance. Then, in that new window it throws an error: "Load failed for F:/Users/Artur/1232. No such file or directory" (last number is random). It gets worse if I use multiprocessing pools, because then Houdini gets even more confused and opens even more of its windows.

I could probably rewrite the script to use dictionaries (hopefully) or push the collected data for processing outside of Houdini, but it would be more convenient (read: requiring less additional work...) to have the job done inside the application. Hence I am curious if someone was able to find a workaround and managed to get Python multiprocessing module up and running in H.

Link to comment
Share on other sites

You realize that the Python interpreter is *NOT* a separate process in Houdini, right? Since it's part of the same process as Houdini, the multiprocessing module can *only* fork the Houdini process. Since you need to export the data out of Houdini anyhow, then you should probably just do what you want outside of Houdini. If you're doing something inherently SIMD, Python is the wrong language to use anyhow. Try VEX.

  • Like 1
Link to comment
Share on other sites

You realize that the Python interpreter is *NOT* a separate process in Houdini, right? Since it's part of the same process as Houdini, the multiprocessing module can *only* fork the Houdini process.

Frankly, I didn't notice this, but it explains a lot, Edward.

I spawned a Hython subprocess (with hou module support) as Houdini's child process, from which I can now use multiprocessing. To communicate between child process and Houdini I use RPC.

After some initial testing it seems to be working fine, so I guess the problem is solved, unless something breaks.

Since you need to export the data out of Houdini anyhow, then you should probably just do what you want outside of Houdini. If you're doing something inherently SIMD, Python is the wrong language to use anyhow. Try VEX.

I'm still very fresh with VEX.
Link to comment
Share on other sites

After spending a couple of hours with Hython subprocesses and RPC I must admit that this wasn't a good idea. Processing data via rpyc turned out to be quite slow, up to such level that it would probably negate all benefits of fully utilizing multiple cores. I also quickly reached maximum chunk limit when fetching large lists from Houdini. Lastly, I noticed that all exception handles were ignored and led to script being aborted.

Not wanting to loose any more time looking for workarounds, I ended up optimizing the code and running it directly in Houdini's single process.

Link to comment
Share on other sites

Perhaps you might find luck with multithreading, as opposed to multiprocessing?

https://docs.python.org/2/library/threading.html#module-threading

 

I think it will end up the the same problem of global lock python interpreter. 

 

There are number of options to optimize your code even inside Houdini/Python (general idea: avoid calling or creating python objects in a loop), pre-processing in VEX, or using some numerical module as it's mostly probably written as C extension. Also: Cythoninlinecpp, hou.runVEX

 

You may attach some part of the problem here as a benchmark / challenge ;)

 

sie ma,

skk.

  • Like 1
Link to comment
Share on other sites

I think it will end up the the same problem of global lock python interpreter. 

 

There are number of options to optimize your code even inside Houdini/Python (general idea: avoid calling or creating python objects in a loop), pre-processing in VEX, or using some numerical module as it's mostly probably written as C extension. Also: Cythoninlinecpp, hou.runVEX

Cześć Szymonie.

Now I know what I will read while sipping my morning coffee. :)

 

You may attach some part of the problem here as a benchmark / challenge ;)

Well, the problem area will probably be trivial to you Houdini Experts, because it all boils down to a Wavefront OBJ exporter that supports vertex normals and materials. Houdini's default gwavefront executable does not export them. So I wrote my own. However, initially for the sake of simplicity and because at that time I was only a couple of days into Houdini, I decided to use lists instead of dictionaries. Lookup times of lists grew proportionally to their lengths which resulted in extreme export times for highpoly geometries.

After swapping lists with dictionaries, export times dropped like millions of times. It now takes approximately 25 seconds to export a 87.360 poly mesh, or around 5:25 to export 1,400,000 one. Much slower than gwavefront exporter (50 seconds, but without vertex normals and materials check), but not even remotely as slow as that unfortunate version with lists...

I chose Python, because that's one of the languages I feel very comfortable with. I realise that there are faster solutions though.

Link to comment
Share on other sites

VEX if applicable is inherently threaded so that would be the first thing I try. If you must use Python I'd do it outside of Houdini and use concurrent.futures in Python 3 if possible since it's so much more user and developer friendly.

 

https://docs.python.org/3.3/library/concurrent.futures.html

 

Not to mention all the command line argument improvements in Python 3 which make it that much easier to do stand alone scripts.

Link to comment
Share on other sites

  • 3 months later...

Use:

 

import platform

import multiprocessing

if platform.system() == 'Windows':
    multiprocessing.set_executable('C:/Python27/pythonw.exe')
elif platform.system() == 'Linux':
    multiprocessing.set_executable('/usr/bin/python')

 

to use the system python. Beware that Houdini 13/14 is using Python 2.7.5 so you need to install the same version.

 

With this trick I was able to run system python with multiprocessing.

Edited by E. Ozgur Yilmaz
Link to comment
Share on other sites

Use:

 

import platform

import multiprocessing

if platform.system() == 'Windows':

    multiprocessing.set_executable('C:/Python27/pythonw.exe')

elif platform.system() == 'Linux':

    multiprocessing.set_executable('/usr/bin/python')

 

to use the system python. Beware that Houdini 13/14 is using Python 2.7.5 so you need to install the same version.

 

With this trick I was able to run system python with multiprocessing.

I'm not sure you will be able to use HOM in system interpreter, at least you need a licence for that..

Link to comment
Share on other sites

  • 2 years later...

I'm having issues with multiprocessing as well, on Linux, where my child processes don't seem to do any work when the script is run in Houdini.

 

set_executable() doesn't exist on Linux so I'm wondering if there is anything else I can do to try to fix it?

 

-Jon

Link to comment
Share on other sites

  • 11 months later...

I was running into problems with multiprocessing and queue in houdini/python in the past with the hou module.  Talked to Jeff at sidefx and said multithreading/queue the hou module can be unstable or all out not work.  Seems dependent on how the hou module is used in multithreading.  Since Houdini is using GIL for multithreading, you can try to tap into that. (Which I'm currently on the hunt to figure out, haha)
I've gotten multiprocessing and queue to work in shelf tools, just not using the hou module and addressing houdini nodes.

jonp, since you are on linux, you could always launch another terminal from houdini to run a command, like a batch houdini call.  When using the `terminal -e` flag, the terminal should self close when the command is completed.  Or at least does on my xfce os.
Sorry I can't be of more help here.

Edited by Trancor
Link to comment
Share on other sites

  • 3 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...