[IPython-User] launching engines via python

Brent Pedersen bpederse@gmail....
Wed Sep 26 10:02:01 CDT 2012


On Tue, Sep 25, 2012 at 5:30 PM, Ben Edwards <bedwards@cs.unm.edu> wrote:
> I ran into this problem not too long ago. I don't think there is any
> programmatic way to do this within iPython, but you can use subprocess. I've
> added in a signal handler, so if I have to stop my script in the middle of a
> run it automatically shuts down the cluster.
>
>> import subprocess
>> import signal
>> import time
>
>
>> def signal_handler(signal,frame):
>>      print("Shutting Down Cluster")
>>      subprocess.Popen(["ipcluster", "stop", "--quiet"])
>>      sys.exit(0)
>
>
>>
>> signal.signal(signal.SIGINT, signal_handler)
>> subprocess.Popen(["ipcluster",
>>                   "start",
>>                   "--n=" + str(12),
>>                   "--daemonize",
>>                   "--quiet"])
>> time.sleep(5) #time for the cluster to spin up
>> #Code using cluster
>> subprocess.Popen(["ipcluster", "stop", "--quiet"])
>
>
> Hope this helps.
>

Thanks, that is a good idea. I'll clarify my use-case on the chance
that it's not clear.
I'd like to be able to have a notebook running on the head node of our
cluster. And
I'd like to be able to start engines dynamically, submit some work,
and then stop the
engines. Our cluster is busy enough that it's, at best, impolite to
hold a few nodes with
engines that arent in use while I look at plots or aggregate results.
Is this a different use than most people have for parallel ipython? Or
are people using
it like Ben suggests?
thanks,
-Brent




> On Tue, Sep 25, 2012 at 1:13 PM, Brent Pedersen <bpederse@gmail.com> wrote:
>>
>> Hi,
>> I'm attempting to understand how I can use an ipengine on an LSF cluster.
>> I've done something like this:
>>
>>      ipython profile create --parallel --profile=lsf
>>      # edit .ipython/profile_lsf/ipcluster_config.py to use LSF
>>      ipcluster start --profile=lsf --daemonize
>>
>> But, I'd like to be able to start the engines programatically.
>> After the above command, I can do this:
>>
>>     c = Client(profile='lsf')
>>
>> Is there a way to "connect" to the profile and have it dynamically launch
>> engines via python?
>> I found the IPEngineApp, but it's not clear to me how to use that.
>>
>> Is there an example somewhere that launches engines via the API, rather
>> than
>> the command line? Even one for PBS would be helpful.
>>
>> thanks,
>> -Brent
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>


More information about the IPython-User mailing list