[IPython-User] Assigning a worker node

Caius Howcroft caius.howcroft@gmail....
Fri Jan 20 16:37:40 CST 2012


Hi

In a lot of our code with have things like this (psuedo code)

def wibble(cfg):
   n = cfg["n"]
   f = openDataset("In%d.txt" %n)
   fo = openDataset("Out%d.txt" %n)

   for i in f:
       fo.write(blah(i))
   fo.close()

lview = LoadBalancedView()
ns = range(100)
cfgs = []

for i in n:
  c = {"n": i,
          ,,,
   }
  cfgs.append(c)


lview.map(wibble, cfgs)


Now that the input has got rather large I want to make sure the same
machine is always assigned for the same N (perhaps even a specific
machine) so I can store things locally. What would be the best way of
doing this? Writing my own chooser function l(like lru, twobin,
weighted ) wont work as those functions are only passed the machine
loads and I need to know what args are being passed to the function
being mapped. Any suggestions?

Thanks in advance

Caius


More information about the IPython-User mailing list