[NumPy-Tickets] [NumPy] #1619: Loading aligned dtype error

[NumPy Trac]

#1619: Loading aligned dtype error
-----------------------------+----------------------------------------------
 Reporter:  Ihor.Melnyk      |       Owner:  somebody    
     Type:  defect           |      Status:  needs_review
 Priority:  highest          |   Milestone:  2.0.0       
Component:  numpy.core       |     Version:  devel       
 Keywords:  dtype alignment  |  
-----------------------------+----------------------------------------------
Changes (by jpeel):

  * status:  new => needs_review


Comment:

 I've come up with a possible solution, but first let me explain the real
 problem here.

 The real problem here is that dtype() can't handle a list that describes
 an aligned dtype. For instance, in the example given above, the list that
 is put into dtype() when the file is loaded is

 {{{[('f0', '|i1'), ('', '|V3'), ('f1', '<i4'), ('f2', '|i1')]}}}

 on my machine. {{{('', '|V3')}}} indicates that this is an aligned dtype.
 The first element in each tuple is the name of the field. When dtype finds
 a name equal to {{{''}}}, it currently sets the name of that field to
 {{{'f#ind'}}} where #ind is the index of the tuple in the list. That means
 that {{{('', '|V3')}}} in the above list has its name set to {{{'f1'}}}
 which then causes an error when the next item has a name of {{{'f1'}}}.

 Okay, so the problem then is that dtype() currently doesn't know how to
 read in an aligned type correctly. Here is how I approached solving this.

 1. If the align parameter doesn't equal 1 and the name of the first tuple
 is not equal to {{{''}}}, then set checkalign to 1 so that this input list
 could indicate an aligned array.

 2. If checkalign, then trigger that a tuple deals with alignment when:
     a. a tuple's name is {{{''}}}
     b. a tuple's title is NULL
     c. a tuple has only two elements
     d. the datatype of the tuple is VOID

 3. When we find a tuple that meets all of the above conditions,
     a. add the size of the VOID tuple to the totalsize
     b. set maxalign to MAX(maxalign, current element size + previous
 element size)
     c. adjust the name index accordingly

 4. Make sure that the alignment is set correctly when checkalign==1.

 5. Readjust the size of the nameslist tuple if necessary. (if there were
 any aligning tuples, then nameslist initialized to be too large).

 3.c. refers to an additional index, ii, to keep track of the indices in
 the nameslist. This is necessary because i refers to the index in the
 input list.

 There might be a better approach than this, but this is one possible
 solution. Other approaches might involve changing the npy format or how
 aligned dtypes are shown. However, I think that those will be much harder
 to implement.

 In the patch that I'm attaching, I've kept everything in the
 _convert_from_array_desc function, but it might be a good idea to break it
 up a bit or make some other changes to make the code more manageable.

-- 
Ticket URL: <http://projects.scipy.org/numpy/ticket/1619#comment:1>
NumPy <http://projects.scipy.org/numpy>
My example project