[IPython-dev] new doc for parallel sessions on clusters

Johann Cohen-Tanugi johann.cohentanugi@gmail....
Tue Jun 21 15:45:48 CDT 2011



On 06/21/2011 07:58 PM, MinRK wrote:
> Glad you were able to fix it.
I am still a bit puzzled by what I see in the 
$HOME/.config/ipython/profile_lsf/ipcluster_config.py
I do not see c.Global, but for instance :
-bash-3.2$ grep engine_launcher ~/.config/ipython/profile_lsf/*
/u/ec/cohen/.config/ipython/profile_lsf/ipcluster_config.py:# 
c.IPClusterStart.engine_launcher_class = 'LSFEngineSetLauncher'
/u/ec/cohen/.config/ipython/profile_lsf/ipcluster_config.py:# 
c.IPClusterEngines.engine_launcher_class = 'LSFEngineSetLauncher'

(where LSF replaces Local as I just edited it)

Johann
> I just pushed an update to the online docs, so they should be up to date now.
thanks a lot. Where is it? 
http://ipython.org/ipython-doc/dev/parallel/parallel_process.html seems 
to still be the old version....

best,
JCT
> On Tue, Jun 21, 2011 at 10:00, Johann Cohen-Tanugi
> <johann.cohentanugi@gmail.com>  wrote:
>> ok found the problem : I still had the ipython-newapp source code in the
>> same dir as the new ipython, I did everything from this new ipython dir,
>> but at execution it was looking into ipython-newapp. Obviously there is
>> something I was not expecting.... Anyway now it works.
>>
>> sorry again,
>> Johann
>>
>> On 06/21/2011 06:41 PM, MinRK wrote:
>>> On Tue, Jun 21, 2011 at 09:31, Johann Cohen-Tanugi
>>> <johann.cohentanugi@gmail.com>    wrote:
>>>> hmmm ok, after the merge is this the right way to get the code :
>>>>   git clone git://github.com/ipython/ipython.git ipython
>>>> I removed the whole source directory, and the local install stuff related
>>>> to
>>>> python, and I started from scratch using the git command above.
>>>> then :
>>>> python setup.py install
>>>> --prefix=/afs/slac/g/glast/users/cohen/IPYDEV/local/which python
>>>>
>>>> perhaps the problem is that there is an ipython installed with the native
>>>> python that I use above, and I am wrong to assume that having
>>>> /afs/slac/g/glast/users/cohen/IPYDEV/local/.../site-packages at the top
>>>> of
>>>> PYTHONPATH is enough to ensure that it overrides the one installed with
>>>> python?
>>> If it was installed with setuptools/easy_install, yes - they inject
>>> packages at the front sys.path at runtime, ending up ahead of
>>> PYTHONPATH. easy_installed packages can even come before '.'.
>>>
>>> You can check which one you are importing by looking at
>>> `IPython.__file__`.
>>>
>>> It's possible that I've messed something up, but the source looks
>>> right
>>> (https://github.com/ipython/ipython/blob/master/IPython/core/profileapp.py#L127)
>>>
>>>> Anyway, I will continue investigating....
>>>> sorry for the noise,
>>>> JOhann
>>>>
>>>> On 06/21/2011 05:27 PM, MinRK wrote:
>>>>> On Tue, Jun 21, 2011 at 07:27, Johann Cohen-Tanugi
>>>>> <johann.cohen-tanugi@lupm.univ-montp2.fr>      wrote:
>>>>>> hi there, I tried to build the sphinx doc on my machine after a pull of
>>>>>> the
>>>>>> head (after Min announced the merge of newapp into master), and I see
>>>>>> the
>>>>>> following in docs/source/parallel/parallel_process.txt :
>>>>>> 1/ A trivial typo at :
>>>>>> :command:`ipcluster` has a notion of Launchers that can start
>>>>>> controllers
>>>>>> and engines with various remote execution schemes.  Currently supported
>>>>>> models include :command:`ssh`, :command`mpiexec`, PBS-style (Torque,
>>>>>> SGE),
>>>>>> and Windows HPC Server.
>>>>>>
>>>>>>   |_____>
>>>>>> should be :command:`mpiexec`
>>>>>> 2/ I can read the following lines :
>>>>>>     $ ipython profile create --parallel profile=mpi
>>>>>> or
>>>>>>     $ ipython profile create --parallel profile=pbs
>>>>>>
>>>>>> while ipython profile create does *not* accept a --parallel option
>>>>>> AFAICT,
>>>>>> but rather --cluster or --no-cluster.
>>>>> Then you do not have current master, because the flag is indeed
>>>>> '--parallel'.
>>>>>
>>>>>> best,
>>>>>> Johann
>>>>>>
>>>>>> On 06/21/2011 07:37 AM, Min RK wrote:
>>>>>>> Yes, it is going to be standard for the cluster profile and
>>>>>>> interactive
>>>>>>> IPython to be different - the profile for the Client is principally
>>>>>>> for
>>>>>>> connection info, and the shell profile is for configuring your
>>>>>>> interactive
>>>>>>> environment.  There's no reason to change your interactive config just
>>>>>>> to
>>>>>>> connect to a different cluster.
>>>>>>>
>>>>>>> That said, the default profile of the Client should probably be that
>>>>>>> of
>>>>>>> the current application, not just 'default'.
>>>>>>>
>>>>>>> -MinRK
>>>>>>>
>>>>>>> On Jun 20, 2011, at 22:11, Johann
>>>>>>> Cohen-Tanugi<johann.cohen-tanugi@lupm.univ-montp2.fr>        wrote:
>>>>>>>
>>>>>>>> hi- I don't think we should print the profile name in the default
>>>>>>>> case,
>>>>>>>>> it's just noise.  I realize we now have a more consistent structure
>>>>>>>>> for profiles and even the default case is now a profile, but we
>>>>>>>>> should
>>>>>>>>> keep the amount of printed stuff to a minimum in the default cases.
>>>>>>>>>
>>>>>>>> Actually I have a question here : I was trying newapp, following
>>>>>>>> Min's
>>>>>>>> advice, to try to add the LSF support in parallel.apps. From what I
>>>>>>>> could
>>>>>>>> gather
>>>>>>>> I did
>>>>>>>> ipcluster start -p lsf -n 2
>>>>>>>> which created profile_lsf in my $HOME/.ipython directory, but then
>>>>>>>> when
>>>>>>>> I
>>>>>>>> started another terminal window for the ipython session, I typed
>>>>>>>> ipython profile=lsf
>>>>>>>> and this loaded the default profile, so that I had to type :
>>>>>>>>
>>>>>>>> from IPython.parallel import Client
>>>>>>>> c = Client(profile='lsf')
>>>>>>>>
>>>>>>>> so that unless this is a bug or an operator mistake, there seems to
>>>>>>>> be
>>>>>>>> 2
>>>>>>>> 'profiles' in such a use case : the ipython global one, and the
>>>>>>>> parallel lsf
>>>>>>>> one. I find that a bit confusing, and maybe there is a way to merge
>>>>>>>> the
>>>>>>>> 2?
>>>>>>>>
>>>>>>>> best,
>>>>>>>> Johann


More information about the IPython-dev mailing list