[IPython-User] IPython.parallel: passing both *args and **kwargs to some_view.map()

[Will Furnass]

Hi all,

I am trying to 'parallelize' a particle swarm optimisation (PSO) 
library.  I've tried to design the PSO implementation so that it can 
accept objective functions with an arbitrary number of dimensions (each 
dimension is an argument).  A dictionary of other arguments can be 
supplied when instantiating a PSO object to provide additional info 
required for the evaluation of the supplied objective function.  

In a nutshell we have:

set_up_pso(objective_function, num_dimensions, 
dict_of_objective_func_args_not_to_be_optimised)

run_pso(matrix_of_floats_for_params_to_optimise):
    for each array in rows(matrix_of_floats):
        some_number = objective_function(*array, 
**dict_of_objective_func_args_not_to_be_optimised)

For the purpose of describing my problem my code looks something like the 
following (ignore 'evaluate_obj_func_parallel' for now):

### module A ###
import numpy as np

class MyOptimiser:
    def __init__(self, some_function, opt_args_dict):
        self.some_function, self.opt_args_dict = some_function, 
opt_args_dict
        self.partial_function = lambda *args : some_function(*args, 
**opt_args_dict)
        self.vectorized_partial_function = np.vectorize
(self.partial_function)
        
    def evaluate_obj_func(self, data_matrix):
        self.results = self.vectorized_partial_function(*data_matrix.T)
        return self.results
        
    def evaluate_obj_func_parallel(self, data_matrix, parallel_view):
        self.results = parallel_view.map(self.some_function, 
*data_matrix.T, **self.opt_args_dict)
        return self.results

### interactive session ###
def my_func(pos1, pos2, pos3, opt1 = 1000, opt2 = 100):
    return pos1 + pos2 + pos3 + opt1 + opt2
my_opt_args={'opt1':5000, 'opt2':500}
my_data_matrix = np.arange(27).reshape(9,3)

my_optimiser = MyOptimiser(my_func, my_opt_args)
my_optimiser.evaluate_obj_func(my_data_matrix)

Out[57]: array([5503, 5512, 5521, 5530, 5539, 5548, 5557, 5566, 5575])

My question is: how can I handle opt_args when using the map method of an 
IPython.parallel.client.view.LoadBalancedView instance?  They're ignored 
if I do the following:

from IPython.parallel import Client
parallel_client = Client()
parallel_view = parallel_client.load_balanced_view()
parallel_view.block = True

my_optimiser.evaluate_obj_func_parallel(my_data_matrix, parallel_view)

Out[58]: [1103, 1112, 1121, 1130, 1139, 1148, 1157, 1166, 1175] # WRONG

and I can't pass lambdas such as partial_function to the engines as 
closures can't be pickled.  Anyone got any good ideas as to how I can 
feed in opt_args when using a parallel map?  

Apologies if I'm missing something obvious here; this is my first foray 
into IPython.parallel.

Cheers,

Will