[Numpy-discussion] Database with Nulls to Numpy Structure

Mark Janikas mjanikas@esri....
Fri Oct 2 18:31:33 CDT 2009

Thanks for the input!  I wonder if I can resize my own record array?  I.e. one call to truncate... Ill give it a go.  But the resize works great as it doesn't make a copy:

In [12]: a = NUM.arange(10)

In [13]: id(a)
Out[13]: 190182896

In [14]: a.resize(5,)

In [15]: a
Out[15]: array([0, 1, 2, 3, 4])

In [16]: id(a)
Out[16]: 190182896

Whereas the slice seems to make a copy/reassign:

In [18]: a = a[0:2]

In [19]: id(a)
Out[19]: 189981184

Pretty Nice.  Pre-allocate the full space and count number of good records... then resize.  Doesn't seem that much faster than using the lists then creating arrays, but memory should be better.  Thanks again, and anything further would be appreciated.


-----Original Message-----
From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Christopher Barker
Sent: Friday, October 02, 2009 12:34 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Database with Nulls to Numpy Structure

Mark Janikas wrote:
> So, do I use lists and 
> append then create the arrays... Or do I fill up the pre-allocated "empty" 
> arrays and slice off the ends?  Thoughts?  Thanks much...

Either will work. I think the decision would be based on how many Null 
records you expect -- if it's a small fraction then go ahead and 
pre-allocate the array, if it's a large fraction, then you might want to 
go with a list.

Note: you may be able to use arr.resize() to chop it off at the end.

The list method has the downside of using more memory, and being a bit 
slower, which may be mitigated if there are lots of null records.

See an upcoming email of mine for another option...


Christopher Barker, Ph.D.

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

NumPy-Discussion mailing list

More information about the NumPy-Discussion mailing list