[IPython-User] ipython parallelism returning user-defined types

Brian Granger ellisonbg@gmail....
Wed Nov 10 11:33:55 CST 2010


John,

On Tue, Nov 9, 2010 at 2:18 AM, John Reid <j.reid@mail.cryst.bbk.ac.uk>wrote:

> Hi,
>
> I'm having some problems returning named tuples from a multi-engine
> client. I do something like:
>
> from IPython.kernel import client
> mec = client.MultiEngineClient()
>
> @mec.parallel()
> def parallel_get_starts(arg):
>     <do some work>
>     return <some named tuples>
>
> results = parallel_get_starts(<some list>)
>
>
> but I always get errors like:
> TypeError: __new__() takes exactly 8 arguments (2 given)
>

Hmm, that is an odd error that I have never seen.  Even the fact that it is
a TypeError is odd to me.  A couple of things to look at:

* Make sure that named tuples can be pickled and unpickled.
* See if you can come up with a super simple example that displays this
error (isolate it).
* See if you can figure out where the TypeError is coming from.




> whenever the results of the parallel_get_starts() function contain named
> tuples. I'm using a namedtuple as given by Raymond Hettinger in the
> Python Cookbook:
> http://code.activestate.com/recipes/500261-named-tuples/
> See below.
>
> Does anyone know how I might get it to work? presumably I could move to
> python 2.6 and use the collections.namedtuple type.
>
>
It would be interesting to see if that shows the error as well.  I am
wondering if the namedtuple implementation you are using has problems.

Cheers,

Brian


> Thanks,
> John.
>
>
>
> def namedtuple(typename, field_names, verbose=False):
>     """Returns a new subclass of tuple with named fields.
>
>     >>> Point = namedtuple('Point', 'x y')
>     >>> Point.__doc__                   # docstring for the new class
>     'Point(x, y)'
>     >>> p = Point(11, y=22)             # instantiate with positional
> args or keywords
>     >>> p[0] + p[1]                     # indexable like a plain tuple
>     33
>     >>> x, y = p                        # unpack like a regular tuple
>     >>> x, y
>     (11, 22)
>     >>> p.x + p.y                       # fields also accessable by name
>     33
>     >>> d = p._asdict()                 # convert to a dictionary
>     >>> d['x']
>     11
>     >>> Point(**d)                      # convert from a dictionary
>     Point(x=11, y=22)
>     >>> p._replace(x=100)               # _replace() is like
> str.replace() but targets named fields
>     Point(x=100, y=22)
>
>     """
>
>     # Parse and validate the field names.  Validation serves two purposes,
>     # generating informative error messages and preventing template
> injection attacks.
>     if isinstance(field_names, basestring):
>         field_names = field_names.replace(',', ' ').split() # names
> separated by whitespace and/or commas
>     field_names = tuple(field_names)
>     for name in (typename,) + field_names:
>         if not min(c.isalnum() or c=='_' for c in name):
>             raise ValueError('Type names and field names can only
> contain alphanumeric characters and underscores: %r' % name)
>         if _iskeyword(name):
>             raise ValueError('Type names and field names cannot be a
> keyword: %r' % name)
>         if name[0].isdigit():
>             raise ValueError('Type names and field names cannot start
> with a number: %r' % name)
>     seen_names = set()
>     for name in field_names
>         if name.startswith('_'):
>             raise ValueError('Field names cannot start with an
> underscore: %r' % name)
>         if name in seen_names:
>             raise ValueError('Encountered duplicate field name: %r' % name)
>         seen_names.add(name)
>
>     # Create and fill-in the class template
>     numfields = len(field_names)
>     argtxt = repr(field_names).replace("'", "")[1:-1]   # tuple repr
> without parens or quotes
>     reprtxt = ', '.join('%s=%%r' % name for name in field_names)
>     dicttxt = ', '.join('%r: t[%d]' % (name, pos) for pos, name in
> enumerate(field_names))
>     template = '''class %(typename)s(tuple):
>         '%(typename)s(%(argtxt)s)' \n
>         __slots__ = () \n
>         _fields = %(field_names)r \n
>         def __new__(cls, %(argtxt)s):
>             return tuple.__new__(cls, (%(argtxt)s)) \n
>         @classmethod
>         def _make(cls, iterable, new=tuple.__new__, len=len):
>             'Make a new %(typename)s object from a sequence or iterable'
>             result = new(cls, iterable)
>             if len(result) != %(numfields)d:
>                 raise TypeError('Expected %(numfields)d arguments, got
> %%d' %% len(result))
>             return result \n
>         def __repr__(self):
>             return '%(typename)s(%(reprtxt)s)' %% self \n
>         def _asdict(t):
>             'Return a new dict which maps field names to their values'
>             return {%(dicttxt)s} \n
>         def _replace(self, **kwds):
>             'Return a new %(typename)s object replacing specified
> fields with new values'
>             result = self._make(map(kwds.pop, %(field_names)r, self))
>             if kwds:
>                 raise ValueError('Got unexpected field names: %%r' %%
> kwds.keys())
>             return result \n\n''' % locals()
>     for i, name in enumerate(field_names):
>         template += '        %s = property(itemgetter(%d))\n' % (name, i)
>     if verbose:
>         print template
>
>     # Execute the template string in a temporary namespace
>     namespace = dict(itemgetter=_itemgetter)
>     try:
>         exec template in namespace
>     except SyntaxError, e:
>         raise SyntaxError(e.message + ':\n' + template)
>     result = namespace[typename]
>
>     # For pickling to work, the __module__ variable needs to be set to
> the frame
>     # where the named tuple is created.  Bypass this step in
> enviroments where
>     # sys._getframe is not defined (Jython for example).
>     if hasattr(_sys, '_getframe'):
>         result.__module__ = _sys._getframe(1).f_globals['__name__']
>
>     return result
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20101110/54fcb3f8/attachment.html 


More information about the IPython-User mailing list