[SciPy-dev] parallelizing cKDTRee

[Sturla Molden]

On 1/7/2009 4:55 PM, Anne Archibald wrote:

> This would be a good idea. One labor-saving device would be to take
> advantage of the fact that construction is pretty cheap and have the
> pickle method pickle only the construction parameters and the
> underlying array. Not that pickling the tree structure would be very
> hard either. 

Yes. I have an example of that in the Cookbook.

I'm actually looking at adding __reduce__ and __setstate__ now. I'll 
post the code tomorrow or on friday.


>> 3. Multithreading (Cython):
>> We could simply call cKDTree.__query in parallel using OpenMP pragmas.
>> It would be a simple and quite portable hack.
> 
> Are there compilation issues with using OpenMP? Otherwise this should
> be fairly easy too, although selecting the right degree of parallelism
> may be a problem (I have no experience with OpenMP).

Yes. There are compilers that don't support it (but gcc does). And there 
is the Cyton to C compilation, which tempers with the variable names. 
The pragma must thus be added to the C code and possibly use variable 
names present in the autogenerated C. Since there is no need for any 
synchronization among OpenMP threads here, I guess this should suffice:


#pragma omp parallel \
private(__pyx_v_c, __pyx_v_k, __pyx_v_dd, \
__pyx_v_ii, __pyx_v_xx, \ __pyx_v_self, \
__pyx_v_eps, __pyx_v_p, \
__pyx_v_distance_upper_bound)


It should be placed right before the line that says:

   for (__pyx_v_c = 0; __pyx_v_c < __pyx_8; __pyx_v_c+=1) {

     /* "/usr/data/david/src/dsp/scipy/trunk/scipy/spatial/ckdtree.pyx":583




Regards,
Sturla Molden