[IPython-user] Trivial parallelisation

Brian Granger ellisonbg.net@gmail....
Mon Dec 1 13:00:25 CST 2008


Yes, IPython will handle this beautifully.  Here is an example, with
everything done interactively:

pcp025387pcs:~ bgranger$ ipython
Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: files = ['file%i.txt' % i for i in range(10)]

In [2]: files
Out[2]:
['file0.txt',
 'file1.txt',
 'file2.txt',
 'file3.txt',
 'file4.txt',
 'file5.txt',
 'file6.txt',
 'file7.txt',
 'file8.txt',
 'file9.txt']

In [3]: from IPython.kernel import client

# this won't load balance, but has low overhead
In [4]: mec = client.MultiEngineClient()

In [5]: def doit(file):
   ...:     pass
   ...:

In [6]: mec.map(doit,files)
Out[6]: [None, None, None, None, None, None, None, None, None, None]

# this does load balance, but has higher overhead
In [7]: tc = client.TaskClient()

In [8]: tc.map(doit,files)
Out[8]: [None, None, None, None, None, None, None, None, None, None]

Our documentation has more details on how to use all of this:

http://ipython.scipy.org/doc/rel-0.9.1/html/parallel/index.html

But, please don't hesitate to ask us questions.

Cheers,

Brian


On Mon, Dec 1, 2008 at 9:50 AM, Jose Gómez-Dans <jgomezdans@gmail.com> wrote:
> Hi,
> I do have some trivial parallel code (I need to apply some analysis algorithm
> to a bunch of data files), and access to either an SMP linux machine with
> loads of CPUs or to a linux cluster. In the past, I have launched my python
> scripts with a different command line to get them to run concurrently, but I
> was thinking whether there's some pythonic way of dealing with this that is
> simple.
>
> I seem to remember that ipython would be a nice tool for this, but I'm not
> sure of it, or indeed, any other alternatives.
>
> My code is simple:
> for file in list_of_files:
>  #I want spawn this next line
>  ProcessFile ( file )
>
> Thanks for your help,
> Jose
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>


More information about the IPython-user mailing list