Python */-&^ Houdini *### Mac *&*?>: Scipy *sigh*

[Macha]

I'm in need of a clever and good soul that knows how to get scipy working with houdini, on a mac. I got numpy working, but not scipy and I know it's because of that stupid Mac's-own-flavour of Python 2.6.

Has anybody been able to compile a version that works?

I'm so desperate, I'm even calling Python2.7 from inside 2.6 now (scipy works in 2.7, of course)...

A good few beers for anybody who can help me.!

Edited October 22, 2012 by Macha

[Erik_JE]

Is there any specific reason for why you can't just install the "real" 2.6 from python.org?

Otherwise http://stackoverflow.com/questions/11442970/numpy-and-scipy-for-preinstalled-python-2-6-7-on-mac-os-lion seems to provide a way

[Macha]

Hey Erik, I've tried that and it messed a few things up. Also, as far as I can work out, Houdini on a Mac insists on using the Mac version of Python.

It also needs to play with numpy as well.

Edited October 22, 2012 by Macha

[lukeiamyourfather]

Not sure about making Houdini use another version of Python, that's probably a question for SESI support. For Python, SciPy, and NumPy on Mac check out MacPorts, it has all of them and will compile, install, and update them for you. Kind of like a package manager and repository you'd find on a Linux machine but it uses source code instead of complied packages.

http://www.macports.org/

Edited October 22, 2012 by lukeiamyourfather

[Macha]

Thanks Luke, I'll go and forage in MacPorts.

In the meantime, below is a piece of scaffolding code that runs scipy in Python2.7 when called from Python2.6. It solves a simple linear algebra problem as a demonstration but works on larger data as well. A 10 million component vector can be passed and retrieved in a few seconds.

At the moment it has to be modified in the case more than 2 arrays are passed. I tried writing by using a function with *args, but that didn't work yet. Perhaps in combination with eval or something like that I can pass around arbitrary amounts of objects. It would probably work by pickling lists containing numpy arrays, but it would be slower than pure numpy writing.

This should also illustrate how to tackle the problem with other modules that can't be accessed in Python2.6

#-----Python2.7 Scipy processor-----
import sys
from scipy import *
import pickle
import time
t=time.clock()

#-----read in parms (eg path) from caller---
filepath = sys.argv[1]

#-----functions to read and write data------
def readDataNumpy(infile ):
	 import numpy as np
	 return np.load(infile)

def writeAwayNumpy(path_to_data, data_to_write):
			 import numpy as np
			 np.savez(path_to_data, data_to_write)

#-----process data--------------------------
def processData(data_npz):
	 from scipy import linalg
	 a = data_npz['arr_0']
	 b = data_npz['arr_1']
	 result = linalg.solve(a,
	 return result

#-----read data-----------------------------
data_npz = readDataNumpy(filepath)
processed = processData(data_npz)

#-----write back result----
writeAwayNumpy(filepath, processed )

print "called script time",time.clock()-t
[/CODE]

and the second file which calls the above script

[CODE]
#-----Python2.6 without Scipy capability-----
from subprocess import *
import time

#-----initialize some variables...---------
t=time.clock()
filepath = '/Users/mp/Documents/data/temp.npz'

#-----...state the problem-----
import numpy as np
a = np.array([[3,2,0],[1,-1,0],[0,5,1]])
b = np.array([2,4,-1])

#-----define functions------------------
def sendToPy(command, pathToScript = '/Users/mp/Documents/data/processScipy.py',
pyVer = "python2.7" ):
commandargs = [pyVer, pathToScript, command]
proc = Popen( commandargs , stdout=PIPE)
return proc.communicate()[0]

def writeAwayNumpy(outfile, a, b ):
import numpy as np
np.savez(outfile, a, b )

def readBackNumpy(filepath):
import numpy as np
return np.load(filepath)

def writeAny(fn, *args):
fn( args[0], args[1] )

#-----write data to file----------------
writeAwayNumpy(filepath, a, b )

#-----call other file-------------------
print sendToPy(filepath)

#-----read back result------------------
print "readback",readBackNumpy(filepath)['arr_0']

print "\ncaller script time",time.clock()-t
[/CODE]

Edited October 23, 2012 by Macha

[Macha]

For multiple objects this function works now:

def writeAwayNumpy(outfile, *args ):
    import numpy as np
    np.savez(outfile, *args )
[/CODE]

Edited October 23, 2012 by Macha

[Macha]

I compiled from source for 2.6, and while I can import the module it does not let me use it...

It's rare but sometimes Houdini just sucks...argh...

Edited October 24, 2012 by Macha

[Erik_JE]

I would rather blame this on OSX

Edited October 24, 2012 by Erik_JE

[Macha]

Yeah...both! I can actually run it in the os, in py2.6 and 2.7 but not in Houdini's preferred 'weird' version.

[Macha]

I made some progress with this.

It seems that when scipy is compiled with gcc44 I can load and use at least some of its functionality in Houdini. I still get a lot of unusual errors when running the official tests but at least some simple things I have tested so far work.

[Macha]

So, I experimented with some frequency filtering on volumes with all this.

If anybody is into that sort of thing, you might find my vimeo post interesting: