[IPython-user] Missing Modules on Ipengines

Hao Xiong hao@biostat.ucsf....
Thu Feb 19 18:13:36 CST 2009


Hi Brian,

Attached are three files, which show the behaviors I encountered.

caller.py uses MultiEngineClient's map method to generate random
numbers in parallel; the actual generating function is rfcn in callee.py.
The random module is imported within callee.py at the file level.
The exception raised is that "'random' is not defined." Moving the import
statement inside function rfcn makes everything work.
The log file is the traceback.

Regards,
Hao


Brian Granger wrote:
> Hao,
>
> Can you send a super simple example that shows this behavior?  That
> will help in debugging it.
>
> Thanks,
>
> Brian
>
> On Thu, Feb 19, 2009 at 11:13 AM, Hao Xiong <hao@biostat.ucsf.edu> wrote:
>   
>> Hi,
>>
>> I am using ipengine in IPython to run multiple instances of a program.
>> The program follows map-reduce pattern where it processes an input and
>> writes results to files within its own directory. Right now I am testing it
>> on one machine with two engines and I get very weired errors.
>>
>> The program is split into multiple files, one of which, "cluster.py",
>> create
>> a MultiEngineClient and calls an imported function with map method.
>> The imported function, "main," is in "para.py." There is also a class
>> called "Options" in
>> "para.py" file, and whose constructor is called inside the imported
>> function. The error I get
>> complains about undefined "Options" variable. What is even more odd is that
>> when I tried to add a print statement to sys.stderr (useless, I know),
>> the exception
>> becomes unknown variable sys, even though sys is imported at the top of
>> "para.py."
>> Adding an import statement inside "main" makes the print statement good.
>>
>> I think this has to do with ipengine's environment, its path, but I
>> cannot be sure.
>> I tried to make a simple test case using two separate files but the
>> simple case works.
>> So right now I am at a loss. I would like some help on my problem and I
>> have some
>> general questions about the inner working of ipengine.
>>
>> 1. Is there a way to connect pdb to ipengine in order to inspect the stack.
>> 2. What exactly is the environment of ipengine? I am concerned with working
>>     directory, path, PYTHONPATH, site modules. This is really import when
>>     a program is split into multiple files, needs calling external
>> programs, and
>>     when outputs go to files.
>> 3. What happens when ipengine needs to call an external program through
>> subprocess
>>     module? What about relative path to regular files?
>>
>> Thanks for any help.
>>
>> Regards,
>> Hao
>> _______________________________________________
>> IPython-user mailing list
>> IPython-user@scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>>
>>     

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: callee.py
Url: http://lists.ipython.scipy.org/pipermail/ipython-user/attachments/20090219/c7e62aec/attachment.pl 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: caller.py
Url: http://lists.ipython.scipy.org/pipermail/ipython-user/attachments/20090219/c7e62aec/attachment-0001.pl 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: log.txt
Url: http://lists.ipython.scipy.org/pipermail/ipython-user/attachments/20090219/c7e62aec/attachment.txt 


More information about the IPython-user mailing list