[Numpy-discussion] NumPy re-factoring project

Pauli Virtanen pav@iki...
Sun Jun 13 07:26:33 CDT 2010


Sun, 13 Jun 2010 06:54:29 +0200, Sturla Molden wrote:
[clip: memory management only in the interface]

You forgot views: if memory management is done in the interface layer, it 
must also make sure that the memory pointed to by a view is never moved 
around, and not freed before all the views are freed.

This probably cannot be done with existing Python objects, since IIRC, 
they always own their own memory. So one should refactor the memory 
management out of ndarray on the interface side.

The core probably also needs to decide when to create views (indexing), 
so a callback to the interface is needed.

> Second: If we cannot figure out how much to allocate before starting to
> use C (very unlikely), 

The point is that while you in principle can figure it out, you don't 
*want to* do this work in the interface layer. Computing the output size 
is just code duplication.

> we can always use a callback to Python. Or we can

This essentially amounts to having a

	NpyArray_resize

in the interface layer, which the core can call.

Here, however, one has the requirement that all arrays needed by core are 
always passed in to it, including temporary arrays. Sounds a bit like 
Fortran 77 -- it's not going to be bad, if the number of temporaries is 
not very large.

If the core needs to allocate arrays by itself, this will unevitably lead 
to memory management that cannot avoided by having the allocation done in 
the interface layer, as the garbage collector must not mistake a 
temporary array referenced only by the core as unused.

An alternative here is to have an allocation routine

	NpyArray_new_tmp
	NpyArray_del_tmp

whose allocated memory is released automatically, if left dangling, after 
the call to the core is completed. This would be useful for temporary 
arrays, although then one would have to be careful not ever to return 
memory allocated by this back to the caller.

> have two C functions, the first determining the amount of allocation,
> the second doing the computation.

That sounds like a PITA, think about LAPACK.

-- 
Pauli Virtanen



More information about the NumPy-Discussion mailing list