[IPython-User] notebooks and clusters

MinRK benjaminrk@gmail....
Tue Jul 24 17:21:44 CDT 2012


When you specify command-line options to the notebook server, via the
command-line or ipython_notebook_config.py, this has no effect on the
ipcluster scripts at all.  You *must* use the ipcluster/ipcontroller
config to configure those scripts.

On Tue, Jul 24, 2012 at 3:07 PM, Darren Govoni <darren@ontrenet.com> wrote:
> I see. Well, what I notice is that the notebook server creates the config
> files when the cluster is started and deletes it when it stops.
> They don't exist when the notebook server is run, only after a cluster is
> started.

Sorry, there is some confusion here.  The JSON files in
profile/security are *not* configuration files, they are connection
files.  These files were written by ipcontroller, which was started as
a subprocess of the notebook server.  The configuration files are
ip{controller|engine|cluster}_config.py in your profile, created by
`ipython profile create [name] --parallel`.

>
> How do I control that? It's using an incorrect url: property preventing
> remote engines from connecting (i.e. the notebook server
> is not listening on the correct IP).
>
> Also, if I do not supply --ip=* as in "ipython notebook --ip=*" then I
> cannot access the notebook remotely. So that option is required for me.
> However, even with this setting the url in the config is incorrect for
> remote engine access.

I did not mean to imply that it is wrong to set the IP of the notebook
server, only that setting the IP of the notebook server has no effect
on the configuration of the Controller (a wholly other process, that
can in fact be on a wholly other machine).

Please read the docs on configuring the ip/ports of the Controller,
where it describes how to configure this (specifically HubFactory.ip
in ipcontroller_config.py):
http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#general-considerations.
 Starting the cluster from the notebook changes nothing about what you
must configure, or how you may specify that configuration.

>
>
> On Mon, 2012-07-23 at 17:09 -0700, Min RK wrote:
>
> Command-line options to the notebook server have exactly zero effect on the
> cluster config. You *must* use the config files to configure the cluster.
>
> -MinRK
>
>
> On Jul 23, 2012, at 16:29, Darren Govoni <darren@ontrenet.com> wrote:
>
>
>
> Hi,
>   I am trying to connect a remote engine to a running notebook using the
> default config. I copied the file.
> It contains.
> {
>   "url": "tcp://127.0.0.1:57726",
>   "exec_key": "6dac7baf-45c3-4ab2-b642-bfa65fdac0e9",
>   "ssh": "",
>   "location": "10.32.6.3"
> }
>
>
> The problem is that the url property is not using the --ip= address I invoke
> the notebook with and
> the remote engine says its timing out trying to register. Any clues?
>
> thanks.
>
> On Mon, 2012-07-23 at 13:52 -0700, MinRK wrote:
>
> On Mon, Jul 23, 2012 at 1:38 PM, Darren Govoni <darren@ontrenet.com> wrote:
>> Ok, thanks.
>>
>> What I want to do is run remote engines manually and have them connect to
>> the notebooks controller.
>>
>> I assume the notebook as a config file it creates that I can use on remote
>> servers for engines to connect?
>
> There is no notebook-specific config to the clusters.  Clicking the
> `start` button for profile 'foo' should be identical to doing
> `ipcluster start --profile foo`.  All of the parallel
> connection/configuration docs still apply.  That means that the
> controller creates `ipcontroller-{client|engine}.json` files in
> profile/security, and you can start engines with `ipengine
> --profile=foo` or `ipcluster engines --profile=foo` to connect to it.
> If your machines are not on a shared filesystem, you will need to pass
> the connection files around as described in the docs.
>
>
>
>>
>> Or does it just use the default config location (e.g. user directory)?
>>
>>
>> On Mon, 2012-07-23 at 13:37 -0700, MinRK wrote:
>>
>> On Mon, Jul 23, 2012 at 1:25 PM, Darren Govoni <darren@ontrenet.com>
>> wrote:
>>> Hi,
>>>   Can the notebook cluster connect with remote engines or vice-versa?
>>
>> Yes - the notebook cluster panel is just a very thin wrapper for
>> ipcluster.  So if you create a profile that starts engines via
>> PBS/SGE/SSH, etc. that will happen when you start a cluster from the
>> panel.  Similarly, you can start a cluster from the panel and add
>> engines to it with `ipengine` and/or `ipcluster engines` in all the
>> usual ways.  There is nothing special about clusters started from the
>> notebook panel, other than that they can also be stopped from there.
>>
>> The one thing you cannot do yet is actually edit the config files for
>> those profiles from the notebook interface - for that, you need access
>> to the filesystem, or you can do a dirty version by writing the the
>> config files with the `%%file` magic in a notebook.
>>
>> -MinRK
>>
>>>
>>> thanks.
>>>
>>> _______________________________________________
>>> IPython-User mailing list
>>> IPython-User@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>


More information about the IPython-User mailing list