[IPython-user] Missing Modules on Ipengines

Vishal Vatsa vishal.vatsa@gmail....
Fri Feb 20 09:04:01 CST 2009


Hi Hao,

I think I see your problem,

You are in effect sending the rfcn method for remote execution but it
has a reference
to random which has not been imported on the ipengine.
Also it might help to think in terms of, you are packaging every thing
needed to execute
on the remote node.

So if you just redo the rfcn method as:
def rfcn(seed):
   import random
   random.seed(seed)
   return random.random()


it should be fine.
Hope that makes sense, if not give me a shout and I will try to explain further.

Regards,
-vishal

2009/2/20 Hao Xiong <hao@biostat.ucsf.edu>:
> Hi Brian,
>
> Attached are three files, which show the behaviors I encountered.
>
> caller.py uses MultiEngineClient's map method to generate random
> numbers in parallel; the actual generating function is rfcn in callee.py.
> The random module is imported within callee.py at the file level.
> The exception raised is that "'random' is not defined." Moving the import
> statement inside function rfcn makes everything work.
> The log file is the traceback.
>
> Regards,
> Hao
>
>
> Brian Granger wrote:
>>
>> Hao,
>>
>> Can you send a super simple example that shows this behavior?  That
>> will help in debugging it.
>>
>> Thanks,
>>
>> Brian
>>
>> On Thu, Feb 19, 2009 at 11:13 AM, Hao Xiong <hao@biostat.ucsf.edu> wrote:
>>
>>>
>>> Hi,
>>>
>>> I am using ipengine in IPython to run multiple instances of a program.
>>> The program follows map-reduce pattern where it processes an input and
>>> writes results to files within its own directory. Right now I am testing
>>> it
>>> on one machine with two engines and I get very weired errors.
>>>
>>> The program is split into multiple files, one of which, "cluster.py",
>>> create
>>> a MultiEngineClient and calls an imported function with map method.
>>> The imported function, "main," is in "para.py." There is also a class
>>> called "Options" in
>>> "para.py" file, and whose constructor is called inside the imported
>>> function. The error I get
>>> complains about undefined "Options" variable. What is even more odd is
>>> that
>>> when I tried to add a print statement to sys.stderr (useless, I know),
>>> the exception
>>> becomes unknown variable sys, even though sys is imported at the top of
>>> "para.py."
>>> Adding an import statement inside "main" makes the print statement good.
>>>
>>> I think this has to do with ipengine's environment, its path, but I
>>> cannot be sure.
>>> I tried to make a simple test case using two separate files but the
>>> simple case works.
>>> So right now I am at a loss. I would like some help on my problem and I
>>> have some
>>> general questions about the inner working of ipengine.
>>>
>>> 1. Is there a way to connect pdb to ipengine in order to inspect the
>>> stack.
>>> 2. What exactly is the environment of ipengine? I am concerned with
>>> working
>>>    directory, path, PYTHONPATH, site modules. This is really import when
>>>    a program is split into multiple files, needs calling external
>>> programs, and
>>>    when outputs go to files.
>>> 3. What happens when ipengine needs to call an external program through
>>> subprocess
>>>    module? What about relative path to regular files?
>>>
>>> Thanks for any help.
>>>
>>> Regards,
>>> Hao
>>> _______________________________________________
>>> IPython-user mailing list
>>> IPython-user@scipy.org
>>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>>>
>>>
>
>
> #!/usr/bin/env python
>
> import random
> def rfcn(seed):
>    random.seed(seed)
>    return random.random()
>
> if __name__ == "__main__":
>    print rfcn(2)
>
> #!/usr/bin/env python
>
> from IPython.kernel import client
>
> def main():
>    from callee import rfcn
>    mec = client.MultiEngineClient()
>    print mec.map(rfcn, [1,1])
> if __name__ == "__main__":
>    main()
>
> Traceback (most recent call last):
>  File "./caller.py", line 10, in <module>
>    main()
>  File "./caller.py", line 8, in main
>    print mec.map(rfcn, [1,1])
>  File
> "//usr/lib64/python2.5/site-packages/IPython/kernel/multiengineclient.py",
> line 813, in map
>    return self.mapper().map(func, *sequences)
>  File "//usr/lib64/python2.5/site-packages/IPython/kernel/mapper.py", line
> 124, in map
>    targets=self.targets, block=self.block)
>  File
> "//usr/lib64/python2.5/site-packages/IPython/kernel/multiengineclient.py",
> line 800, in raw_map
>    dist, targets=targets, block=block)
>  File
> "//usr/lib64/python2.5/site-packages/IPython/kernel/multiengineclient.py",
> line 441, in _blockFromThread
>    result = blockingCallFromThread(function, *args, **kwargs)
>  File "//usr/lib64/python2.5/site-packages/IPython/kernel/twistedutil.py",
> line 69, in blockingCallFromThread
>    return twisted.internet.threads.blockingCallFromThread(reactor, f, *a,
> **kw)
>  File "//usr/lib64/python2.5/site-packages/twisted/internet/threads.py",
> line 83, in blockingCallFromThread
>    result.raiseException()
>  File "//usr/lib64/python2.5/site-packages/twisted/python/failure.py", line
> 319, in raiseException
>    raise self.type, self.value, self.tb
> IPython.kernel.error.CompositeError: one or more exceptions from call to
> method: execute
> [0:execute]: NameError: global name 'random' is not defined
> [1:execute]: NameError: global name 'random' is not defined
>
> _______________________________________________
> IPython-user mailing list
> IPython-user@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>
>


More information about the IPython-user mailing list