[IPython-User] IPython.parallel: Sync namespace between engines

[Francesco Montesano]

2013/1/29 Johann Rohwer <jr@sun.ac.za>

> On Tuesday 29 January 2013 13:37:34 Florian M. Wagner wrote:
> > I am currently trying to parallelize a piece of code using IPython's
> > parallel version of map. It works fine for simple examples, but the
> > function I would like to distribute falls back upon some globally
> > declared variables, which the engines are not aware of. Is there a
> > way to sync the namespace of all engines analogously to
> > sync_imports or what would you suggest?
>
> I'm not aware of whether it is possible to sync the namespaces
> completely and whether this useful or feasible at all (do we want the
> controller and client objects to be available on all the engines?).
> However, you can easily push those global variables the function
> requires to all the engines with the dview.push() method.
>
> Johann
> E-pos vrywaringsklousule
>
> Hierdie e-pos mag vertroulike inligting bevat en mag regtens
> geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit
> geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee
> in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of
> kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in
> kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid
> vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of
> die oopmaak van enige lês aangeheg by hierdie e-pos nie.
>
> E-mail disclaimer
>
> This e-mail may contain confidential information and may be legally
> privileged and is intended only for the person to whom it is addressed. If
> you are not the intended recipient, you are notified that you may not use,
> distribute or copy this document in any manner whatsoever. Kindly also
> notify the sender immediately by telephone, and delete the e-mail. The
> University does not accept liability for any damage, loss or expense
> arising from this e-mail and/or accessing any files attached to this e-mail.
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
Hi Florian,

I asked something related a few month ago: my main problem was to `import
module as m` and broadcast functions and variables to all the engines. As
now that is no automatic way.

So I wrote a class that initialize my parallel environment, and perform a
few operations. One of the methods is

    def exec_on_engine( self, code, block=True ):
        """
        Execute the given code on all engines

        Parameters
        ----------
        to_execute: string or list of strings
            command(s) to execute on all the nodes. Thought for short tasks,
            like importing modules
        block: bool
            whether or not to wait until done to return. default: True
        """
        if isinstance(code, basestring): # if it's a string
            code = [code,]  # convert to list
        # execute the required commands
        # (better to do in block mode, avoids errors if command is slow)
        for te in code:
            try:
                self.dview.execute(te, block=block)
            except error.CompositeError, e:  # if an error occurs, print a
single one, not one per engine
                e.raise_exception()


I use it roughly as follow (I didn't test passing a variable, but should
work)

        import os

        # absolute path and file name of this script

        path, fname = os.path.split(os.path.abspath(sys.argv[0]))

        #command to run on all the engines

        imports = ['import numpy as np',
                # add the script directory to the python path

                'import sys', 'sys.path.append("{0}")'.format(path),

                # import a function from the script in the namespace

                'from {0} import func'.format(os.path.splitext(fname)[0]),
                 'a = 10']
        parallel_env.exec_on_engine(imports)

Does it help?

Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130129/40c2814e/attachment.html