[SciPy-user] histogramdd question

David Huard david.huard@gmail....
Thu Aug 16 08:58:54 CDT 2007


2007/8/16, Chris Lee <c.j.lee@tnw.utwente.nl>:
>
> Thanks David,
>
> That makes sense. How difficult would it be to put in an optional
> parameter to only use integers?


Not difficult at all, in fact, but I'm guessing there are not many people
around with the kind of needs you have ! I can't even begin to understand
how a computer can handle a 700G object. In any case, my suggestion would be
for you to copy and paste the histogramdd fonction from numpy and tweak it
to you heart's content. Better yet for such large arrays would be for you to
rewrite the function in fortran and wrap it using f2py. I'd also advise
using a "bloc" approach, where you feed the histogram function chunks of the
array instead of the whole thing. This avoids swapping memory to the hard
drive when you exceed you RAM capacity. If you want to give it a try, I'll
send you some drafts to get you started.

Cheers,
David

In some sense this doesn't matter too much since I can (in pathological
> cases) end up with a 700 GB histogram but normally It clocks in between
> 0.8-2 GB. In the end type casting isn't going to stop me from grabbing
> all the memory I can.
>
>
>
> David Huard wrote:
> > Hi Chris,
> >
> > histogramdd will consist of floats when it is normalized or when
> > non-integer weights are given. I thought it was preferable to return
> > floats in all cases so the output is consistent no matter what. It
> > could be changed if there is a compelling argument, but I'd rather not
> > risk the chance of breaking someone's code.
> >
> > Cheers,
> >
> > David
> >
> > 2007/8/15, Chris Lee <c.j.lee@tnw.utwente.nl
> > <mailto:c.j.lee@tnw.utwente.nl>>:
> >
> >     Hi All,
> >
> >     I have been using histogramdd to generate histograms of a 4-D data
> >     set.
> >     A histogram should, I believe, return integers but histogramdd
> returns
> >     doubles. Is there are reason for this?
> >
> >     Cheers
> >     Chris
> >
> >     --
> >     **********************************************
> >     *  Chris Lee                                 *
> >     *  Laser physics and nonlinear optics group  *
> >     *  MESA+ Institute                           *
> >     *  University of Twente                      *
> >     *  Phone: ++31 (0)53 489 3968                *
> >     *  fax: ++31 (0) 53 489 1102                 *
> >     **********************************************
> >
> >     _______________________________________________
> >     SciPy-user mailing list
> >     SciPy-user@scipy.org <mailto:SciPy-user@scipy.org>
> >     http://projects.scipy.org/mailman/listinfo/scipy-user
> >
> >
>
> --
> **********************************************
> *  Chris Lee                                 *
> *  Laser physics and nonlinear optics group  *
> *  MESA+ Institute                           *
> *  University of Twente                      *
> *  Phone: ++31 (0)53 489 3968                *
> *  fax: ++31 (0) 53 489 1102                 *
> **********************************************
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user@scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/scipy-user/attachments/20070816/54512b42/attachment.html 


More information about the SciPy-user mailing list