[IPython-user] Missing Modules on Ipengines

Hao Xiong hao@biostat.ucsf....
Fri Feb 20 13:48:18 CST 2009


Thanks, Brian. I think I got it. I made a helper function
that contains all the import statements and pass that function
to map method. Of course, all the files are available locally
to the engines.

Thanks a lot.

Best,
Hao


Brian Granger wrote:
>> The thing is Python has static scoping, so reference to random
>> should have been embedded inside rfcn. To further highlight this,
>> I changed the import code to
>>     
>
> Yes, the reference to random is in rfcn, but when the engine tries to
> resolve the reference, it finds that random has not been imported
> (only the function is sent to the engines, not referenced modules).
>
>   
>> import caller
>> mec.map(callee.rfcn, [1,1])
>>     
>
> This doesn't work because it only imports random on the client, not
> the engines.  This would work though:
>
> mec.execute('import random')
> mec.map(callee.rfcn, [1,1])
>
>   
>> For my real program, moving import statement inside the function
>> is impractical because it looks like this:
>>
>> class Options():
>>    ...
>>
>> def main():
>>    opt = Options()
>>    ...
>>
>>     
>
> That is OK, then just use mec.execute to pre-import your module before
> using its functions.  The engine is just like a regular python
> interpreter in this respect.
>
> Or you can put all the initialization code in a local file and have
> the engine execute all of it using run:
>
> mec.run('init.py') # copies local file init.py to engines and runs it
>
> Cheers,
>
> Brian
>
>   
>> The function main is then called through map method of
>> MultiEngineClient. Besides there are other files eventually
>> called by and through main,  so the problem exists on a large scale.
>>
>> Thanks for you help.
>>
>> Best,
>> Hao
>>
>>
>> Vishal Vatsa wrote:
>>     
>>> Hi Hao,
>>>
>>> I think I see your problem,
>>>
>>> You are in effect sending the rfcn method for remote execution but it
>>> has a reference
>>> to random which has not been imported on the ipengine.
>>> Also it might help to think in terms of, you are packaging every thing
>>> needed to execute
>>> on the remote node.
>>>
>>> So if you just redo the rfcn method as:
>>> def rfcn(seed):
>>>    import random
>>>    random.seed(seed)
>>>    return random.random()
>>>
>>>
>>> it should be fine.
>>> Hope that makes sense, if not give me a shout and I will try to explain further.
>>>
>>> Regards,
>>> -vishal
>>>
>>> 2009/2/20 Hao Xiong <hao@biostat.ucsf.edu>:
>>>
>>>       
>>>> Hi Brian,
>>>>
>>>> Attached are three files, which show the behaviors I encountered.
>>>>
>>>> caller.py uses MultiEngineClient's map method to generate random
>>>> numbers in parallel; the actual generating function is rfcn in callee.py.
>>>> The random module is imported within callee.py at the file level.
>>>> The exception raised is that "'random' is not defined." Moving the import
>>>> statement inside function rfcn makes everything work.
>>>> The log file is the traceback.
>>>>
>>>> Regards,
>>>> Hao
>>>>
>>>>
>>>>
>>>>         
>> _______________________________________________
>> IPython-user mailing list
>> IPython-user@scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
>>
>>     



More information about the IPython-user mailing list