[Numpy-discussion] Numpy 2D array from a list error
Wed Sep 23 11:56:20 CDT 2009
Appologies for the multiple posts, people. My posting to the forum was
pending for a long time, so I deleted it and tried emailing directly. I
didn't think they'd all be sent out.
Gokan, thanks for the reply, I hope you get this one.
"Here I use loadtxt to read ~89 MB txt file. Can you use loadtxt and share
I: data = np.loadtxt('09_03_18_07_55_33.sau', dtype='float',
I: len data
I: len data
Variable Type Data/Info
data ndarray 66x117040: 7724640 elems, type `float64`, 61797120
bytes (58 Mb)
[gsever@ccn various]$ python sysinfo.py
Python : ('CPython', 'tags/r26', '66714')
IPython : 0.10
NumPy : 1.4.0.dev
Matplotlib : 1.0.svn
I tried using loadtxt and got the same error as before (with a little more
Traceback (most recent call last):
File "/home/dwood/workspace/GeneralScripts/src/test_clab2R.py", line 140,
File "/home/dwood/workspace/GeneralScripts/src/test_clab2R.py", line 45,
data = loadtxt("inputfile.txt",dtype='string')
505, in loadtxt
X = np.array(X, dtype)
ValueError: setting an array element with a sequence
Thanks for the information. To fix my problem, I tried taking out the row
names (leaving only numerical information), and converting the 2D list to
floats. I still had the same problem.
On 9/23/09, Christopher Barker <Chris.Barker@noaa.gov> wrote:
> Dave Wood wrote:
> > Well, I suppose they are all considered to be strings here. I haven't
> > tried to convert the numbers to floats yet.
> This could be an issue. For strings, numpy creates an array of strings,
> all of the same length, so each element is as big as the largest one:
> In : l
> Out: ['5', '34', 'this is a much longer string']
> In : np.array(l)
> array(['5', '34', 'this is a much longer string'],
> Note that each element is 28 bytes (that's what the S28 means).
> this means that your array would be much larger than the text file if
> you have even one long string it in. Also, as mentioned in this thread,
> in order to figure out how big to make each string element, the array()
> constructor has to scan through your entire list first, and I don't know
> how much intermediate memory it may use in that process.
> This really isn't how numpy is meant to be used -- why would you want a
> big ol' array of mixed numbers and strings, all stored as strings?
> structured arrays were meant for this, and np.loadtxt() is the easiest
> way to get one.
> > I just tried preallocating the array and updating it one line at a time,
> > and that works fine.
> what dtype do you end up with?
> > This doesn't seem like the expected behaviour though and the error
> > message seems wrong.
> yes, not a good error message at all -- it's hard to make sure good
> errors get triggered every time!
> Christopher Barker, Ph.D.
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
> NumPy-Discussion mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the NumPy-Discussion