[Numpy-discussion] pickling/unpickling numpy.void and numpy.record for multiprocessing

Pauli Virtanen pav@iki...
Fri Feb 26 17:26:00 CST 2010


pe, 2010-02-26 kello 14:41 -0800, Martin Spacek kirjoitti:
[clip: pickling/unpickling numpy.void scalar objects]
> I suppose numpy.void is as it suggests, a pointer to a specific place in memory. 
> I'm just surprised that this pointer isn't dereferenced before pickling Or is 
> it? I'm not skilled in interpreting the strings returned by pickle.dumps(). I do 
> see the word "Hello" in the string, so maybe the problem is during unpickling.

No, the unpickled void scalar will own its data. The problem is that
either the data is not saved correctly (unlikely), or it is unpickled
incorrectly.

The relevant code path to look at is multiarraymodule:array_scalar ->
scalarapi.c:PyArray_Scalar. Needs some cgdb'ing to find out what's going
on there.

Please file a bug report on this.

> Is this a known limitation? Any suggestions on how I might get around this? 
> Pool.map() pickles each numpy.void entry as it iterates over the structured 
> array, before sending it to the next available process. 

Use 1-element arrays instead of void scalars. Those will pickle
correctly. Perhaps reshaping your array to (N, 1) will be enough.

-- 
Pauli Virtanen



More information about the NumPy-Discussion mailing list