[Numpy-discussion] Create numpy array from a list error
Gökhan Sever
gokhansever@gmail....
Wed Sep 23 11:24:51 CDT 2009
On Wed, Sep 23, 2009 at 9:06 AM, Dave Wood <davejwood@gmail.com> wrote:
> Hi all,
>
> I've got a fairly large (but not huge, 58mb) tab seperated text file, with
> approximately 200 columns and 56k rows of numbers and strings.
>
> Here's a snippet of my code to create a numpy matrix from the data file...
>
> ####
>
> data = map(lambda x : x.strip().split('\t'), sys.stdin.readlines())
> data = array(data)
>
> ###
>
> It causes the following error:
>
> data = array(data)
> ValueError: setting an array element with a sequence
>
> If I take the 1st 40,000 lines of the file, it works fine.
> If I take the last 40,000 lines of the file, it also works fine, so it
> isn't a problem with the file.
>
> I've found a few other posts complaining of the same problem, but none of
> their fixes work.
>
> It seems like a memory problem to me. This was reinforced when I tried to
> break the dataset into 3 chunks and stack the resulting arrays - I got an
> error message saying "memory error".
> Also, I don't really understand why reading in this 57mb txt file is taking
> up ~2gb's of RAM.
>
> Any advice? Thanks in advance
>
> Dave
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
Here I use loadtxt to read ~89 MB txt file. Can you use loadtxt and share
your results?
I[14]: data = np.loadtxt('09_03_18_07_55_33.sau', dtype='float',
skiprows=83).T
I[15]: len data
-----> len(data)
O[15]: 66
I[16]: len data[0]
-----> len(data[0])
O[16]: 117040
I[17]: whos
Variable Type Data/Info
--------------------------------
data ndarray 66x117040: 7724640 elems, type `float64`, 61797120
bytes (58 Mb)
[gsever@ccn various]$ python sysinfo.py
================================================================================
Platform :
Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
Python : ('CPython', 'tags/r26', '66714')
IPython : 0.10
NumPy : 1.4.0.dev
Matplotlib : 1.0.svn
================================================================================
--
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20090923/93514911/attachment.html
More information about the NumPy-Discussion
mailing list