[SciPy-dev] parallelizing cKDTRee

Sturla Molden sturla@molden...
Wed Jan 7 11:54:29 CST 2009


On 1/7/2009 6:39 PM, Sturla Molden wrote:

>> This would be a good idea. One labor-saving device would be to take
>> advantage of the fact that construction is pretty cheap and have the
>> pickle method pickle only the construction parameters and the
>> underlying array. Not that pickling the tree structure would be very
>> hard either. 
> 
> Yes. I have an example of that in the Cookbook.
> 
> I'm actually looking at adding __reduce__ and __setstate__ now. I'll 
> post the code tomorrow or on friday.


Oh, I did not think of that :)

I pickled the whole kd-tree, but it would be even cheaper to just have 
the worker processes build their own from the data. If the workers equal 
the number of processors, the extra overhead will be zero. Which means 
there would be no need for supporting the pickle protocol at all.

Sturla Molden





More information about the Scipy-dev mailing list