[IPython-dev] DAG Dependencies

MinRK benjaminrk@gmail....
Thu Oct 28 14:44:04 CDT 2010


On Thu, Oct 28, 2010 at 12:34, Brian Granger <ellisonbg@gmail.com> wrote:

> On Thu, Oct 28, 2010 at 12:00 PM, Fernando Perez <fperez.net@gmail.com>
> wrote:
> > On Thu, Oct 28, 2010 at 11:55 AM, MinRK <benjaminrk@gmail.com> wrote:
> >> My interactive session starts running into memory issues with 100M ids,
> so
> >> that's something to consider.
> >
> > Well, let's just hope that anyone with 100M ids in real life (not a
> > synthetic benchmark) has the common sense of working on something with
> > 100G of RAM or so :)
>
> But, with the performance we are getting, some people might want to
> simply run this 24/7 (instead of using a batch system).  Then, over a
> long period of time, having that many tasks is not too crazy.  We
> definitely need to think about this aspect of things.
>

Until I put in the DB backend, the memory footprint of the Controller after
100M tasks is going to be huge, so we can worry about that a little later.
 Possibly add a sense of msg_ids going stale?  Only keep the most recent 10M
for checking?

-MinRK


>
> > Thanks a lot for looking into this, Min.  Fantastic example to have,
> > and one that is very relevant to many use cases.
>
> Definitely,
>
> Cheers,
>
> Brian
>
> > Cheers,
> >
> > f
> >
>
>
>
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger@calpoly.edu
> ellisonbg@gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-dev/attachments/20101028/33b74df6/attachment-0001.html 


More information about the IPython-dev mailing list