<br><br><div class="gmail_quote">On Sun, Feb 12, 2012 at 11:48, Darren Govoni <span dir="ltr"><<a href="mailto:darren@ontrenet.com">darren@ontrenet.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Sun, 2012-02-12 at 11:12 -0800, MinRK wrote:<br>
><br>
><br>
> On Sun, Feb 12, 2012 at 10:42, Darren Govoni <<a href="mailto:darren@ontrenet.com">darren@ontrenet.com</a>><br>
> wrote:<br>
> Thanks Min,<br>
><br>
> Is it possible to open a ticket for this capability for a<br>
> (near) future<br>
> release? It compliments that already amazing load balancing<br>
> capability.<br>
><br>
><br>
> You are welcome to open an Issue. I don't know if it will make it<br>
> into one of the next few releases, but it is on my todo list. The<br>
> best way to get this sort of thing going is to start with a Pull<br>
> Request.<br>
<br>
</div>Ok, I will open an issue. Thanks. In the meantime, is it possible for<br>
clients to 'know' when a controller is no longer available? For example,<br>
it would be nice if I can insert a callback handler for this sort of<br>
internal exception so I can provide some graceful recovery options.<br></blockquote><div><br></div><div>It would be sensible to add a heartbeat mechanism on the controller->client PUB channel for this information. Until then, your main controller crash detection is going to be simple timeouts.</div>
<div><br></div><div>ZeroMQ makes disconnect detection a challenge (because there are no disconnect events, because a disconnected channel is still valid, as the peer is allowed to just come back up).</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
><br>
><br>
> Perhaps a related but separate notion would be the ability to<br>
> have<br>
> clustered controllers for HA.<br>
><br>
><br>
> I do have a model in mind for this sort of thing, though not multiple<br>
> *controllers*, rather multiple Schedulers. Our design with 0MQ would<br>
> make this pretty simple (just start another scheduler, and make an<br>
> extra call to socket.connect() on the Client and Engine is all that's<br>
> needed), and this should allow scaling to tens of thousands of<br>
> engines.<br>
<br>
</div>Yes! That's what I'm after. In this cloud-scale age of computing, that<br>
would be ideal.<br>
<br>
<br>
Thanks Min.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
><br>
> On Sun, 2012-02-12 at 08:32 -0800, Min RK wrote:<br>
> > No, there is no failover mechanism. When the controller<br>
> goes down, further requests will simply hang. We have almost<br>
> all the information we need to bring up a new controller in<br>
> its place (restart it), in which case the Client wouldn't even<br>
> need to know that it went down, and would continue to just<br>
> work, thanks to some zeromq magic.<br>
> ><br>
> > -MinRK<br>
> ><br>
> > On Feb 12, 2012, at 5:02, Darren Govoni<br>
> <<a href="mailto:darren@ontrenet.com">darren@ontrenet.com</a>> wrote:<br>
> ><br>
> > > Hi,<br>
> > > Does ipython support any kind of clustering or failover<br>
> for<br>
> > > ipcontrollers? I'm wondering how situations are handled<br>
> where a<br>
> > > controller goes down when a client needs to perform<br>
> something.<br>
> > ><br>
> > > thanks for any tips.<br>
> > > Darren<br>
> > ><br>
> > > _______________________________________________<br>
> > > IPython-User mailing list<br>
> > > <a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
> > > <a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
> > _______________________________________________<br>
> > IPython-User mailing list<br>
> > <a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
> > <a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
><br>
><br>
> _______________________________________________<br>
> IPython-User mailing list<br>
> <a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
><br>
><br>
> _______________________________________________<br>
> IPython-User mailing list<br>
> <a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
<br>
<br>
_______________________________________________<br>
IPython-User mailing list<br>
<a href="mailto:IPython-User@scipy.org">IPython-User@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-user" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-user</a><br>
</div></div></blockquote></div><br>