[Numpy-discussion] Advice on converting iterator into array efficiently

Alan Jackson alan@ajackson....
Fri Aug 29 21:22:29 CDT 2008


I tested all three offered solutions :

t = table[:] # convert to structured array
collections = np.unique(t['collection'])
for collection in collections:
    cond = t['collection'] == collection
    energy_this_collection = t['energy'][cond]
----------------------------------

energies = {}
for row in table:
    c = row['collection']
    e = row['energy']
    if c in energies:
        energies[c].append(e)
    else:
	energies[c] = [e]

# Convert the lists in numpy arrays
for key in energies:
    energies[key] = numpy.array(energies[key])
---------------------------------

for c in np.unique(table.col('collection')) :
    print c,' : ', table.readWhere('collection == c', field='energy')

and the timing results were rather dramatic :

time 1 =  0.79
time 2 =  0.08
time 3 =  10.35

This was a test on a relatively small table. I'll have to try it out on
something really big next and see how the memory usage works out.

Thanks guys!

- Alan

-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan@ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


More information about the Numpy-discussion mailing list