[IPython-user] usage of Task's depend

[John Swinbank]

Hi Brian,

On 8 Apr 2009, at 18:10, Brian Granger wrote:

> The current design of the task dependency system and properties has a
> number of flaws and unfortunately, you are running into all of them.
>
> But, I think I have good news.  I think I have figured out a way of
> handling this that is much more robust and doesn't have the
> limitations the current approach does.

Thanks a lot for this! Very useful.

The only thing that makes me a little uncomfortable is this:

> * The task will then fail if the dependencies are not met.  To get the
> task to reschedule itself, just specify a number of task retries.  You
> will just need to give sufficient retries that the task will
> eventually be run on an engine that has the dependency.

Presumably, however many retries are given, there's always a  
possibility that this might not happen -- I guess there's some chance  
that the task will always be sent back to the same engine (I don't  
think the retry system is smart enough to ensure it goes somewhere  
else next time... or am I wrong?). Obviously, with enough retries the  
odds of hitting a useful engine become pretty good, but it seems like  
the whole system might get bogged down with tasks bouncing around all  
over the place, specially when the number of tasks and engines gets  
large.

Very interesting to get your feedback, anyway. I'll try implementing  
something along the lines you suggest and see how well it works, and  
keep an eye open for your future developments.

Cheers,

John