[IPython-User] IPython-User Digest, Vol 94, Issue 42

Jesus Aguillon jesus@aguillon....
Mon Aug 15 17:48:16 CDT 2011


Ill be home about 830 after class.
On Aug 15, 2011 1:55 PM, <ipython-user-request@scipy.org> wrote:
> Send IPython-User mailing list submissions to
> ipython-user@scipy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mail.scipy.org/mailman/listinfo/ipython-user
> or, via email, send a message with subject or body 'help' to
> ipython-user-request@scipy.org
>
> You can reach the person managing the list at
> ipython-user-owner@scipy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of IPython-User digest..."
>
>
> Today's Topics:
>
> 1. Disabling auto-docstring in qtconsole (Ariel Rokem)
> 2. Re: Disabling auto-docstring in qtconsole (Thomas Kluyver)
> 3. Re: Disabling auto-docstring in qtconsole (MinRK)
> 4. Running Embedded IP (Darren Govoni)
> 5. Re: Running Embedded IP (Thomas Kluyver)
> 6. IPython wiki (Thomas Kluyver)
> 7. Re: Running Embedded IP (Darren Govoni)
> 8. Pushing Code to Engines? (Darren Govoni)
> 9. Re: Pushing Code to Engines? (MinRK)
> 10. Re: ipcluster in ssh mode - (MinRK)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 15 Aug 2011 10:10:42 -0700
> From: Ariel Rokem <arokem@gmail.com>
> Subject: [IPython-User] Disabling auto-docstring in qtconsole
> To: ipython-user@scipy.org
> Message-ID:
> <CA+X4w0zW1OEdfhPbptB-5Mtfi3R6bQ1V5qqXm+VZabpNz5KBkQ@mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Hi everyone,
>
> Is there any way to configure the qtconsole such that it *does not*
> show the docstring of an object when I type a left parentheses after
> it?
>
> Thanks,
>
> Ariel
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 15 Aug 2011 18:22:02 +0100
> From: Thomas Kluyver <takowl@gmail.com>
> Subject: Re: [IPython-User] Disabling auto-docstring in qtconsole
> To: Ariel Rokem <arokem@gmail.com>
> Cc: ipython-user@scipy.org
> Message-ID:
> <CAOvn4qjHd7xM_Fo_R-TD+EjOJ-03X7zX4ANr1j3eV_64JNgCxQ@mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On 15 August 2011 18:10, Ariel Rokem <arokem@gmail.com> wrote:
>
>> Is there any way to configure the qtconsole such that it *does not*
>> show the docstring of an object when I type a left parentheses after
>> it?
>
>
> Not yet. There's an issue open for it:
>
> https://github.com/ipython/ipython/issues/694
>
> Thanks,
> Thomas
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/c9eebd42/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Mon, 15 Aug 2011 10:22:22 -0700
> From: MinRK <benjaminrk@gmail.com>
> Subject: Re: [IPython-User] Disabling auto-docstring in qtconsole
> To: Ariel Rokem <arokem@gmail.com>
> Cc: ipython-user@scipy.org
> Message-ID:
> <CAHNn8BVUg8p79S-TcgDPH-FVPuw=Qfcfk0cdor+2a094tZV1MQ@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Not configurable yet, but on the todo list:
>
> https://github.com/ipython/ipython/issues/694
>
> -MinRK
>
> On Mon, Aug 15, 2011 at 10:10, Ariel Rokem <arokem@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Is there any way to configure the qtconsole such that it *does not*
>> show the docstring of an object when I type a left parentheses after
>> it?
>>
>> Thanks,
>>
>> Ariel
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/e3fb35fd/attachment-0001.html
>
> ------------------------------
>
> Message: 4
> Date: Mon, 15 Aug 2011 13:59:15 -0400
> From: Darren Govoni <darren@ontrenet.com>
> Subject: [IPython-User] Running Embedded IP
> To: ipython-user@scipy.org
> Message-ID: <4E495E73.9010005@ontrenet.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi,
> I have an embedded IPython shell in my python app.
> How can I submit to it some commands, python code to run (as a string)?
>
> ip = InteractiveShellEmbed()
>
> I viewed the online docs for it, but it didn't show how to do this
> programmatically.
>
> thanks,
> Darren
>
>
> ------------------------------
>
> Message: 5
> Date: Mon, 15 Aug 2011 19:11:41 +0100
> From: Thomas Kluyver <takowl@gmail.com>
> Subject: Re: [IPython-User] Running Embedded IP
> To: Darren Govoni <darren@ontrenet.com>
> Cc: ipython-user@scipy.org
> Message-ID:
> <CAOvn4qhUhVeBvrNtFfBGUEvMGrZ3dKueV_yq-o9nN5zZBA2qdw@mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On 15 August 2011 18:59, Darren Govoni <darren@ontrenet.com> wrote:
>
>> ip = InteractiveShellEmbed()
>>
>> I viewed the online docs for it, but it didn't show how to do this
>> programmatically.
>>
>
> To run pure Python code in the interactive namespaces, "ip.ex(code)"
should
> work. To run code as if it was entered at the prompt (translating magic
> functions, saving as a history entry, increasing prompt count), use
> "ip.run_cell(code)". In the latter case, code needs to be a string, not a
> compiled code object.
>
> There may also be a way using config to run code as the shell starts up,
but
> I'm not sure if that works for embedding.
>
> Thomas
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/3c11a54a/attachment-0001.html
>
> ------------------------------
>
> Message: 6
> Date: Mon, 15 Aug 2011 20:27:29 +0100
> From: Thomas Kluyver <takowl@gmail.com>
> Subject: [IPython-User] IPython wiki
> To: IPython User list <ipython-user@scipy.org>
> Message-ID:
> <CAOvn4qg95Y=KYf+DHk8-FGNi0uDMRhnUr8xx5+3_vvW6d2rzbA@mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> I noticed our wiki was attracting spam, so I've enabled some simple
> CAPTCHAs, in the form of text questions which should be trivial for Python
> users. These should show up when you create a new page, create an account,
> or add links to a page, which I hope balances convenience and protection
> nicely.
>
> Please let me know if this seems to be causing any problems.
>
> Thanks,
> Thomas
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/94c579c5/attachment-0001.html
>
> ------------------------------
>
> Message: 7
> Date: Mon, 15 Aug 2011 16:33:30 -0400
> From: Darren Govoni <darren@ontrenet.com>
> Subject: Re: [IPython-User] Running Embedded IP
> To: Thomas Kluyver <takowl@gmail.com>
> Cc: ipython-user@scipy.org
> Message-ID: <4E49829A.9060104@ontrenet.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Thank you.
>
>
> On 08/15/2011 02:11 PM, Thomas Kluyver wrote:
>> On 15 August 2011 18:59, Darren Govoni <darren@ontrenet.com
>> <mailto:darren@ontrenet.com>> wrote:
>>
>> ip = InteractiveShellEmbed()
>>
>> I viewed the online docs for it, but it didn't show how to do this
>> programmatically.
>>
>>
>> To run pure Python code in the interactive namespaces, "ip.ex(code)"
>> should work. To run code as if it was entered at the prompt
>> (translating magic functions, saving as a history entry, increasing
>> prompt count), use "ip.run_cell(code)". In the latter case, code needs
>> to be a string, not a compiled code object.
>>
>> There may also be a way using config to run code as the shell starts
>> up, but I'm not sure if that works for embedding.
>>
>> Thomas
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/8d91366d/attachment-0001.html
>
> ------------------------------
>
> Message: 8
> Date: Mon, 15 Aug 2011 16:34:51 -0400
> From: Darren Govoni <darren@ontrenet.com>
> Subject: [IPython-User] Pushing Code to Engines?
> To: ipython-user@scipy.org
> Message-ID: <4E4982EB.9090508@ontrenet.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi,
> I have a cluster and create a Client() from ip shell.
> I define a function in my shell space. Is it possible to have it pushed
> to the engines to execute? Or does all the code have to reside
> on the engines already?
>
> thanks for any tips. I'm reading all the docs, but missing these
> particulars.
>
> Darren
>
>
> ------------------------------
>
> Message: 9
> Date: Mon, 15 Aug 2011 13:51:29 -0700
> From: MinRK <benjaminrk@gmail.com>
> Subject: Re: [IPython-User] Pushing Code to Engines?
> To: Darren Govoni <darren@ontrenet.com>
> Cc: ipython-user@scipy.org
> Message-ID:
> <CAHNn8BUpsG6tSNCeHXGhmgUw-dOAEQQ6e0swtYwBeKpDk_6Ttw@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> To call a function on args remotely, you just use apply:
>
> rc = Client()
> dview = Client[:]
>
> def f(a):
> return a*2
>
> # call f(5) on every engine:
> dview.apply(f, 5)
>
> # send f to all engines (as 'g', you can give it any name), so you can use
> it many times:
> dview['g'] = f
>
> # now you can call it with execute:
> dview.execute('b=g(10)')
>
> -MinRK
>
> On Mon, Aug 15, 2011 at 13:34, Darren Govoni <darren@ontrenet.com> wrote:
>
>> Hi,
>> I have a cluster and create a Client() from ip shell.
>> I define a function in my shell space. Is it possible to have it pushed
>> to the engines to execute? Or does all the code have to reside
>> on the engines already?
>>
>> thanks for any tips. I'm reading all the docs, but missing these
>> particulars.
>>
>> Darren
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/1ec4bf6e/attachment-0001.html
>
> ------------------------------
>
> Message: 10
> Date: Mon, 15 Aug 2011 13:54:47 -0700
> From: MinRK <benjaminrk@gmail.com>
> Subject: Re: [IPython-User] ipcluster in ssh mode -
> To: Manuel Jung <mjung@astrophysik.uni-kiel.de>
> Cc: ipython-user@scipy.org
> Message-ID:
> <CAHNn8BWHX-q-3DmNYj=5qR65+y76jK7e+Pm5vkdfFeGDvPmzhA@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Fri, Aug 12, 2011 at 03:44, Manuel Jung <mjung@astrophysik.uni-kiel.de
>wrote:
>
>> Hi,
>>
>> So i have been browsing around the sources, looking for another solution,
>> to make my use case built in ipcluster, because i were feeling stupid for
>> writing a script to setup the cluster, if this is, what ipcluster should
do
>> for me.
>>
>
> Honestly, there are many situations for which writing a simple bash script
> or using screen will always be better than ipcluster. Since there are so
> many possible configurations and considerations, a general tool will
always
> be more complicated than one that caters to a particular environment. For
> instance, you should probably create only one set of tunnels per machine,
> rather than per engine, since 16*8 tunnels is *a lot*, and largely
pointless
> (it is entirely workload/system dependent which is preferable). Certainly,
> ipcluster should handle your case better, but that doesn't mean it's the
> ideal tool for you.
>
>
>> So this is my solution:
>>
>> Since we are on a totaly restricted network/pc and ports are never to be
>> reached (execpt for ssh/22) outside of localhost, it is totally save to
>> choose static ports like you suggested in your first post in this thread.
>>
>> c.LocalControllerLauncher.controller_args = ['--log-level=20',
>> '--ip=0.0.0.0', '--location=127.0.0.1', '--port=10101',
>> '--HubFactory.hb=10102,10112', 'HubFactory.control=10203,10103',
>> '--HubFactory.mux=10204,10104', '--HubFactory.task=10205,10105']
>>
>> For tunneling from the engines' host, i have implemented an additional
>> parameter for the SSHEngineSetLauncher. It allows to run a shell command
on
>> the engines' host. In this case it is used to establish all tunnels.
>>
>> tunnel = ['ssh dwarf20 -N -L10101:127.0.0.1:10101 -L10102:127.0.0.1:10102
-L10112:
>> 127.0.0.1:10112 -L10103:127.0.0.1:10103 -L10104:127.0.0.1:10104-L10105:
127.0.0.1:10105
>> '.split()]
>> c.SSHEngineSetLauncher.engines = {'pluto' : (16, None, tunnel),
>> 'merkur' : (4, None,
>> tunnel)}
>>
>> (dwarf20 is the cluster starting client an controller hosting pc, pluto
and
>> merkur servers for number crunching, e.g. engines' hosts.)
>>
>> Let me say at this point, that establishing tunnels for all ports in one
>> command isn't always a good idea, because they share the same tcp
>> connections and bandwidth is restricted on a per connection basis. So
maybe
>> this may be a bottleneck under high load.
>>
>
> I think it is highly unlikely that putting all the traffic of a single
> engine on one tcp connection would be a bottle neck, because under most
> normal usage, there will not be significant traffic on multiple sockets at
> the same time. A case that could bottleneck would be a very large number
of
> very short tasks that print a lot to stdout/err.
>
>
>>
>> Still this is not enough for getting all connections working. On pluto
with
>> 16 cores i experienced often less than 16 successfull connected engines.
I
>> found, that simultaneous authentications to an sshd are restricted to 10
by
>> the MaxStartups parameter (see man sshd_config(5)). So i introduced a new
>> parameter for delaying consecutive ssh connections.
>>
>> c.SSHEngineSetLauncher.delay = 0.2
>>
>
> delay is great, I will actually add it to the LocalEngineSetLauncher,
> because it should even be useful at that level (SSHEngineSetLauncher will
> inherit it).
>
>
>>
>> The complete setup from ipcluster_config.py can be found in the post
>> scriptum. I have created a branch on github for this, see
>> https://github.com/gzahl/ipython/tree/sshenvironment
>>
>> This works for me at the moment, what do you think about this solution.
>>
>
> Thanks for working this out! I think it should be a good starting point.
> Some things should probably change - We don't want to define a class
> n*p-times inside another, and reindenting the code so it doesn't match the
> rest of the file is probably not desirable. It's possible that a simple
> configurable preflight script on the SSHLauncher would be a cleaner
> solution, and provide an avenue for more general customization.
>
>
>> Two last thoughts:
>> - It would be nice, if one wouldn't have to specify the port
configuration
>> an tunnel command explicit. It would be nice if you could only define the
>> ports and activate tunneling=yes. But i'm not sure how this could be done
>> best - yet.
>>
>
> Something like this would definitely be valuable. Note that with my
> enginessh branch, if the Controller was launched with
--enginessh=anything,
> then tunneling *will* be enabled by default (if the `ssh` field of the
JSON
> file is specified, it is used. You can edit it manually after starting the
> controller, if you like).
>
> The problem with just 'tunneling=yes' is that it's extremely variable what
> tunneling will look like. We can support one or two simple cases (like the
> one I cover in enginessh).
>
>
>> - I have to define '--profile=ssh' in the program_args for the
>> SSHEngineLauncher - shouldn't this be automaticly choosen, if i'm
starting
>> with "ipcluster start --profile=ssh"? It seems like a bug to me?
>>
>
> This presumes that the profile exists on the remote machine and that the
> initial profile was specified by name and not by path, which is
> insufficiently general. What should actually happen is to send the
> connection file and use it explicitly, with no assumptions about the
remote
> filesystem, or remote profiles available.
>
>
>> - I were testing with the ControlMaster feature of SSH (version 4 or
>> greater). It reuses a existing tcp connection and can speed up new ssh
>> connections. But one would ran into the only-one-tcp-connection issues
>> again. Do you know this command? I'm not sure if it is of use in this
case.
>> But it could help to lower the SSHEngineSetLauncher.delay parameter.
>>
>
> I am aware of it. Since we can't depend on it, I'm not sure how valuable
it
> is to ipcluster in general (another case where writing against your own
> environment lets you make assumptions that aren't appropriate for
> ipcluster).
>
> In general, the SSH launchers need to be improved. The sshx code in 0.10.2
> was better in many ways, but not in others.
>
>
>>
>> Cheers
>> Manuel
>>
>>
>> ipcluster_config.py:
>>
>> c = get_config()
>> c.IPClusterStart.engine_launcher_class = 'SSHEngineSetLauncher'
>> c.IPClusterStart.delay = 2.0
>> c.LocalControllerLauncher.controller_args = ['--log-level=20',
>> '--ip=0.0.0.0', '--location=127.0.0.1', '--port=10101',
>> '--HubFactory.hb=10102,10112', 'HubFactory.control=10203,10103',
>> '--HubFactory.mux=10204,10104', '--HubFactory.task=10205,10105']
>> # Are hard coded paths really a reasonable default? On my systems this
>> doesn't make much sense.
>>
>
> For local launchers, they absolutely are. This means that the programs
will
> be run from the same Python, etc. as the ipcluster script. Otherwise there
> could be weird situations where 'ipcontroller' launched in a subprocess
> actually points to a different Python or IPython than the one launching it
> (this has happened *many* times).
>
>
>
>> c.LocalControllerLauncher.controller_cmd = ['ipcontroller']
>> c.SSHEngineLauncher.program = ['ipengine']
>> c.SSHEngineLauncher.program_args = ['--log_level=20', '--profile=ssh']
>> c.SSHEngineSetLauncher.engine_args = ['--log-level=20', '--profile=ssh']
>> c.SSHEngineSetLauncher.delay = 0.2
>> tunnel = ['ssh dwarf20 -N -L10101:127.0.0.1:10101 -L10102:127.0.0.1:10102
-L10112:
>> 127.0.0.1:10112 -L10103:127.0.0.1:10103 -L10104:127.0.0.1:10104-L10105:
127.0.0.1:10105
>> '.split()]
>> c.SSHEngineSetLauncher.engines = {'pluto' : (16, None, tunnel),
>> 'merkur' : (4, None,
>> tunnel)}
>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/f54081b9/attachment.html
>
> ------------------------------
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
> End of IPython-User Digest, Vol 94, Issue 42
> ********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20110815/3381c129/attachment-0001.html 


More information about the IPython-User mailing list