[IPython-User] ipcluster, profiles, startup scripts and the web notebook

Matthew Turk matthewturk@gmail....
Fri Jul 27 18:53:57 CDT 2012


Hi Min

On Fri, Jul 27, 2012 at 5:31 PM, MinRK <benjaminrk@gmail.com> wrote:
>
>
> On Fri, Jul 27, 2012 at 2:19 PM, Matthew Turk <matthewturk@gmail.com> wrote:
>>
>> Hi Min,
>>
>> On Fri, Jul 27, 2012 at 4:04 PM, MinRK <benjaminrk@gmail.com> wrote:
>> >
>> >
>> > On Fri, Jul 27, 2012 at 10:53 AM, Matthew Turk <matthewturk@gmail.com>
>> > wrote:
>> >>
>> >> Hi there,
>> >>
>> >> I'm trying to set up a custom profile for IPython running on my local
>> >> machine under MPI, but I'm running into trouble.  This is with the
>> >> released 0.13.  I'd like at startup for a couple commands to be run,
>> >> to signal to my Python package that it should expect to be running in
>> >> parallel (with MPI).  Unfortunately, I'm not quite getting the
>> >> behavior I'd expect, and I think it must be something simple on my end
>> >> that I'm not understanding.
>> >>
>> >> I've created a new profile with:
>> >>
>> >> ipython profile create --profile yt --parallel
>> >>
>> >> Into the startup directory I've placed the normal startup commands --
>> >> in this case, the main import for the module I intend to work with --
>> >> inside a 00-startup.py file.  I'm able to spawn new notebooks that
>> >> start up as I'd expect if I do "ipython notebook --profile=yt".  So
>> >> that seems to work properly.  For starting in parallel, I've modified
>> >> the generated profile in a few places:
>> >>
>> >>  * I've set c.IPClusterEngines.profile = u'yt' in ipcluster_config.py
>> >>  * I've changed c.IPClusterEngines.engine_launcher_class to "MPI" in
>> >> ipcluster_config.py
>> >
>> >
>> > Exactly right.
>> >
>> >>
>> >>  * In ipengine_config.py I've set c.IPEngineApp.profile = u'yt' (I was
>> >> not sure if this was either unnecessary or redundant)
>> >
>> >
>> > Yes, setting profile in config files is indeed always redundant (both
>> > cases,
>> > here).
>> >
>> >>
>> >>  * In ipengine_config.py I've added a startup command in
>> >> c.IPEngineApp.startup_command
>> >
>> >
>> > Re-reading the code, this config was broken when the forked Kernel
>> > implementations were merged for 0.13.
>> > I think right now it is impossible to trigger startup code on engines in
>> > 0.13 or master.
>> > Can you open an Issue for this?
>> >
>> > The reason being that Kernels/Shells are shared, but the Application
>> > objects
>> > are still separate, and that is where this startup config lives.
>> > Ultimately, IPEngineApp should go away, but we aren't quite there yet.
>>
>> Ah, okay.  Thanks for digging into this.  I've opened issue #2213:
>>
>> https://github.com/ipython/ipython/issues/2213
>
>
> Thanks!  And I may have a fix for you.
>

Those changes fixed it for me when applied to 0.13.  Thanks!

-Matt

>>
>>
>>
>> Best,
>>
>> Matt
>>
>> >
>> >>
>> >> In the web notebook launcher, changes to the number of processors is
>> >> reflected in what happens when I click on "start".  And looking at the
>> >> output of top/ps I also see that the engines have been launched with
>> >> mpiexec as I have requested.  I'm also able to connect to them and
>> >> verify that they have been launched, but running "dir()" doesn't show
>> >> the expected imports, from the startup_command.  I'm also unable to
>> >> get anything to be run if I set c.MPI.init_script to be an existing
>> >> script.  Am I missing something with how to set up the configuration?
>> >>
>> >> Thanks for any ideas,
>> >>
>> >> Matt
>> >> _______________________________________________
>> >> IPython-User mailing list
>> >> IPython-User@scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/ipython-user
>> >
>> >
>> >
>> > _______________________________________________
>> > IPython-User mailing list
>> > IPython-User@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-user
>> >
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>


More information about the IPython-User mailing list