[Numpy-discussion] NumPy-Discussion Digest, Vol 47, Issue 61

David Goldsmith d.l.goldsmith@gmail....
Sat Aug 21 17:58:59 CDT 2010


On Sat, Aug 21, 2010 at 10:00 AM, <numpy-discussion-request@scipy.org>wrote:

> Date: Fri, 20 Aug 2010 14:30:58 -0500
> From: Robert Kern <robert.kern@gmail.com>
> Subject: Re: [Numpy-discussion] Making MATLAB and Python "play nice"
> To: Discussion of Numerical Python <numpy-discussion@scipy.org>
> Message-ID:
>        <AANLkTikrwzD0VtjiSK+6xh2dJBCa1v1SxX_LN6g4gjoO@mail.gmail.com<AANLkTikrwzD0VtjiSK%2B6xh2dJBCa1v1SxX_LN6g4gjoO@mail.gmail.com>
> >
> Content-Type: text/plain; charset=UTF-8
>
> On Fri, Aug 20, 2010 at 14:25, David Goldsmith <d.l.goldsmith@gmail.com>
> wrote:
> > Hi! ?Please forgive the re-post: I forgot to change the subject line
> > and I haven't seen a response to this yet, so I'm assuming the former
> > might be the cause of the latter.
>
> Or perhaps because the licenses are plainly visible at the links?
>

Ah, I see: if I can't be bothered to click on the links, no one else can be
bothered to tell me that that's all I need to do to get my question
answered.  Unfortunately, the solutions are useless to me if they're not
freely redistributable, so I have no incentive to click on the links--which
do not advertise that they answer the licensing question--'til that question
is answered - catch-22.

DG


>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ? -- Umberto Eco
>
>
> ------------------------------
>
> Message: 4
> Date: Sat, 21 Aug 2010 00:31:03 +0100
> From: Francesc Alted <faltet@pytables.org>
> Subject: Re: [Numpy-discussion] [ANN] carray: an in-memory compressed
>        data    container
> To: Discussion of Numerical Python <numpy-discussion@scipy.org>
> Message-ID:
>        <AANLkTimUAAhqVGg4XWkTwtv+CW+S_YpaVXTPxnxfa0BZ@mail.gmail.com<AANLkTimUAAhqVGg4XWkTwtv%2BCW%2BS_YpaVXTPxnxfa0BZ@mail.gmail.com>
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> 2010/8/20, Zbyszek Szmek <zbyszek@in.waw.pl>:
> > OK, I've got a case where carray really shines :|
> >
> > zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python
> > bench/concat.py numpy 800000 1000 4 1
> > problem size: (800000) x 1000 = 10^8.90309
> > time for concat: 4.806s
> > size of the final container: 6103.516 MB
> > zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python
> > bench/concat.py concat 800000 1000 4 1
> > problem size: (800000) x 1000 = 10^8.90309
> > time for concat: 3.475s
> > size of the final container: 6103.516 MB
> > zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python
> > bench/concat.py carray 800000 1000 4 1
> > problem size: (800000) x 1000 = 10^8.90309
> > time for concat: 1.434s
> > size of the final container: 373.480 MB
> >
> > Size is set to NOT hit the swap. This is still the easily compressible
> > arange... but still, the results are very nice.
>
> Wow, the results with your processor are much nicer than with my Atom
> indeed.  But yeah, I somewhat expected this because Blosc works much
> faster with recent processors, as can be seen in:
>
> http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks
>
> BTW, the difference between memcpy and memmove times for this
> benchmark is almost 40% for your computer, which is really large :-/
> Hmm, something must go really wrong with memcpy in some glibc
> distributions...
>
> At any rate, for real data that is less compressible the advantages of
> carray will be less apparent, but at least the proof of concept seems
> to work as intended, so I'm very happy with it.  I'm also expecting
> that the combination carray/numexpr would perform faster than plain
> computations programmed in C, most specially with modern processors,
> but will see how much faster exactly.
>
> > Of course when the swap is hit, the ratio between carray and a normal
> array
> > can grow to infinity :)
> >
> > zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python
> > bench/concat.py numpy 1000000 1000 3 1
> > problem size: (1000000) x 1000 = 10^9
> > time for concat: 35.700s
> > size of the final container: 7629.395 MB
> > zbyszek@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python
> > bench/concat.py carray 1000000 1000 3 1
> > problem size: (1000000) x 1000 = 10^9
> > time for concat: 1.751s
> > size of the final container: 409.633 MB
>
> Exactly.  This is another scenario where the carray concept can be
> really useful.
>
> --
> Francesc Alted
>
>
> ------------------------------
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> End of NumPy-Discussion Digest, Vol 47, Issue 61
> ************************************************
>



-- 
Privacy is overrated; Identity isn't.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20100821/3e4e9886/attachment-0001.html 


More information about the NumPy-Discussion mailing list