[SciPy-User] Scipy views and slicing: Can I get a view-slice from only certain elements of an array?

Francesc Alted faltet@pytables....
Tue Nov 2 05:32:52 CDT 2010


A Tuesday 02 November 2010 10:04:12 Jacob Biesinger escrigué:
> > Why not using a numpy.array object instead of array.array?
> >  Indexing
> > 
> > with them is much faster than using plain lists:
> >>>> a = np.arange(1000)
> >>>> b = np.arange(1e8)
> >>>> timeit b[a]
> > 
> > 100000 loops, best of 3: 10.4 µs per loop
> > 
> >>>> l = a.tolist()
> >>>> timeit b[l]
> > 
> > 10000 loops, best of 3: 66.5 µs per loop
> 
> Thanks for pointing this out-- using scipy.array instead of lists of
> ints shaved my iteration time from 4 minutes to 2.5 minutes.
> 
> > NumPy arrays helps saving space too:
> >>>> sys.getsizeof(l)
> > 
> > 8072
> > 
> >>>> a.size*a.itemsize
> > 
> > 8000   # 72 bytes less, not a lot but better than nothing
> 
> My data actually looks more like lots of smallish lists.  It looks
> like scipy.array and int lists take up about the same space.
> a = [range(10) for i in xrange(1000000)]  # V=223m RES=182m
> a = [scipy.array(range(10), dtype=scipy.int32) for i in
> xrange(1000000)] # V=275m RES=192m

Yes.  For arrays that small probably the overhead of numpy headers make 
them to take more space than plain lists.  The crosspoint for which 
numpy containers consumes less memory starts is with 100's:

> a = [range(100) for i in xrange(100000)]
Memory usage: ******* list *******
VmSize:  170764 kB      VmRSS:   98832 kB
VmData:   96304 kB      VmStk:     180 kB
VmExe:     1352 kB      VmLib:   10584 kB

>  a = [np.array(range(100), dtype=np.int32) for i in xrange(100000)]
Memory usage: ******* numpy *******
VmSize:  136748 kB      VmRSS:   64964 kB
VmData:   62292 kB      VmStk:     176 kB
VmExe:     1352 kB      VmLib:   10584 kB


-- 
Francesc Alted


More information about the SciPy-User mailing list