[SciPy-dev] parallelizing cKDTRee
Wed Jan 7 11:54:29 CST 2009
On 1/7/2009 6:39 PM, Sturla Molden wrote:
>> This would be a good idea. One labor-saving device would be to take
>> advantage of the fact that construction is pretty cheap and have the
>> pickle method pickle only the construction parameters and the
>> underlying array. Not that pickling the tree structure would be very
>> hard either.
> Yes. I have an example of that in the Cookbook.
> I'm actually looking at adding __reduce__ and __setstate__ now. I'll
> post the code tomorrow or on friday.
Oh, I did not think of that :)
I pickled the whole kd-tree, but it would be even cheaper to just have
the worker processes build their own from the data. If the workers equal
the number of processors, the extra overhead will be zero. Which means
there would be no need for supporting the pickle protocol at all.
More information about the Scipy-dev