[Numpy-discussion] Help using numPy to create a very large multi dimensional array

Nick Fotopoulos nvf@uwm....
Wed Apr 18 18:09:38 CDT 2007


On 4/18/07, numpy-discussion-request@scipy.org
<numpy-discussion-request@scipy.org> wrote:

> ------------------------------
>
> Message: 5
> Date: Wed, 18 Apr 2007 09:11:32 -0700
> From: Christopher Barker <Chris.Barker@noaa.gov>
> Subject: Re: [Numpy-discussion] Help using numPy to create a very
>         large   multi dimensional array
> To: Discussion of Numerical Python <numpy-discussion@scipy.org>
> Message-ID: <46264334.8080304@noaa.gov>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Bruno Santos wrote:
> > Finally I was able to read the data, by using the command you sair with
> > some small changes:
> > matrix = numpy.array([[float(x) for x in line.split()[1:]] for line in
> > vecfile])
>
> it doesn't sound like you're concerned about the speed of reading the
> files, but you can still use fromfile() or maybe fromstring() to do
> this. You just need to read past the text part first, then process it.
>
> using fromstring:
>
> matrix = numpy.vstack([numpy.fromstring(line.split(" ", 1)[1], sep=" ")
> for line in vecfile])
>
> or something like that.
>
> -Chris

I would strongly recommend pylab.load.  It handles comments, selects
columns, and is legible.

Examples from the docstring:

        t,y = load('test.dat', unpack=True) # for  two column data
        x,y,z = load('somefile.dat', usecols=(3,5,7), unpack=True)

A more advanced example from examples/load_converter.py:

dates, closes = load(
    'data/msft.csv', delimiter=',',
    converters={0:datestr2num}, skiprows=1, usecols=(0,2),
    unpack=True)

Devs, is there any possibility of moving/copying pylab.load to numpy?
I don't see anything in the source that requires the rest of
matplotlib.  Among convenience functions, I think that this function
ranks pretty highly in convenience.

Take care,
Nick


More information about the Numpy-discussion mailing list