[Numpy-discussion] re cfunctions help with concatenating (vstack, hstack, etc.)

Skipper Seabold jsseabold@gmail....
Mon Dec 7 18:12:09 CST 2009


On Mon, Dec 7, 2009 at 7:00 PM, John [H2O] <washakie@gmail.com> wrote:
>
> Hello (Pierre?),
>
> I'm trying to work more with structured arrays, which at times seems great,
> and at others (due to my lack of familiarity) very frustrating.
>

Sounds familiar...

> Anyway, right now I'm writing a bit of code to read a series of files with
> x,y,z data. I'm creating record arrays for each file a read. Once I have
> them all read, I just want to load them into one big array. But it doesn't
> seem as straight forward as concatenation:
>

Hmm, I just went through a lot of data cleaning in a similar manner.
I used structured arrays rather than record arrays though, as I
believe the latter are faster to pull from.

> In [102]: type(D); D.shape; D.dtype.names
> Out[102]: <class 'numpy.core.records.recarray'>
> Out[102]: (3025,)
> Out[102]: ('datetime', 'lon', 'lat', 'elv', 'co')
>
> In [103]: type(D2); D2.shape; D2.dtype.names
> Out[103]: <class 'numpy.core.records.recarray'>
> Out[103]: (3445,)
> Out[103]: ('datetime', 'lon', 'lat', 'elv', 'co')
>
> In [104]: C = rf.stack_arrays((D,D2))
>
> In [105]: type(C); C.shape; C.dtype.names
> Out[105]: <class 'numpy.ma.core.MaskedArray'>
> Out[105]: (6470,)
> Out[105]: ('datetime', 'lon', 'lat', 'elv', 'co')
>
> In [106]: C.datetime
> ---------------------------------------------------------------------------
> AttributeError                            Traceback (most recent call last)
>
> /xnilu_wrk/flex_wrk/jfb/RESEARCH_ARCTIC/Arctic_CO/<ipython console> in
> <module>()
>
> AttributeError: 'MaskedArray' object has no attribute 'datetime'
>
> In [107]: C[0]
> Out[107]: (datetime.datetime(2008, 6, 29, 14, 50, tzinfo=<UTC>),
> 248.83900164808117, 53.949661856137709, -0.31834712473582627,
> 112.91218844292422)
>
> In [108]:
>
> So it seems I end up with a masked array of the correct length, but it is an
> array of tuples and no longer a record array. Am I missing a step?
>

Well I think all record arrays are essentially arrays of tuples (or
structs) if I understand correctly.

You can try passing usemask=False and asrecarray=True to stack_arrays.
 I had to play with these when using the recfunctions at times.  I'm
not sure the defaults are consistent, though I didn't really check.

Incidentally, is there a reason that the recfunctions aren't imported
into the np.lib namespace?

Skipper


More information about the NumPy-Discussion mailing list