[SciPy-user] numarray interface and performance issues (for dot product and transpose)

A.Schmolck a.schmolck at gmx.net
Fri Mar 1 15:07:29 CST 2002


Hi Fernando,

Fernando Perez <fperez at pizero.colorado.edu> writes:
> I hope that if this undergraduate project you mention is something serious,
> they could consider joining forces with the scipy plotting effort. One of the

AFAIK it is a serious attempt to provide fully fledged plotting abilities for
python (2D, 3D, images, zooming, saving in ps etc.), but unfortunately, I
suspect that since the shared nature of open source development fits uneasily
with the assessment policy of my university for 3rd year projects, no code
will be publicly released before the end of easter (as much as I myself would
like to have a look at it now and hack around...).


> Well, as others told you using a Matrix instead of an array will get you this
> notation. Here's an example:

I knew about Matrix, but I was concerned that the required casting to use
standard scipy/Numeric functions with "Matrixes" might cause considerable
clutter, but maybe that concern was a bit unfounded...

[snipped]
> Now, I don't know internally how these operations are implemented, they
> probably return copies of their objects (slow, expensive for big
> matrices). But I was wondering if it would be worth considering making .T and
> .cc simple flags. Basically, one could define them via getters (properties in
> python 2.2) so that all the above syntax works, but internally the getter for
> a.T would return a with a flag for transposition on. Then any matrix operation
> could check for whether this flag was on or not and arrange its looping order
> to honor the convention without changing ever the data in a. The same thing
> could be done for complex conjugation.
> 
> This may turn out to be a really stupid idea, but at least I'd like to hear
> other's opinions. This would require all the basic operations on matrices
> (__getitem__, __add__, __mult__, etc) to check for these flags. But once that
> job is done (once), operations involving complex conjugation or transposition
> could be done very fast, only keeping copies of the original data (since both
> a.T and a.cc can be expressed in terms of a). For the .cc case this would
> require writing the basic complex algebra in terms of conjugated numbers, but
> that's a minor annoyance. 
> 
> The tradeoff seems to be one of complexity in the base operations code and a
> bit of extra checking (an if test at the start). But for large arrays this may
> give significant benefits.
> 
> Communication problems: if one needs to pass a.T to an external library which
> has no idea of how to handle these flags, there's a problem. If this idea
> doesn't have other  fundamental flaws (which it probably does), one way around
> this crucial problem would be to have _two_ different ways of getting
> transpose(a) --similar ideas for conjugation:
> 
>  - a.T -> 'flagged' object, only works with libraries which know how to handle
> it but allows writing clean, fast code with zero-copy.
> 
>  - a.Transpose() -> makes a copy and returns a normal array with the elements
> transposed. Any routine which can handle a numpy array can handle this.
> 
Hmm, interestingly you are the only one who picked up on the lazy copying
theme so far, apart from another poster who suggested that no unnecessary
copies are created anyway (I have no time to check that right now,
unfortunately).

I'd certainly much prefer if this work would get done behind the scenes (as
far as I know it isn't but I might be wrong).  I feel Numeric itself should
really handle this transparently if possible, but allowing the user to
explicitly specify that he wants something like transpose and slice copies
done eagerly might be an idea (it shouldn't change the semantics, but might be
worthwhile for debugging purposes or for efficiency/desired "realtime"
behavior in situations where it is clear that an actual copy operation needs
to be performed). Of course talk is cheap and I don't know how much pain such
an implementation would involve, but to me it seems very attractive.

alex

-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.Schmolck at gmx.net     http://www.dcs.ex.ac.uk/people/aschmolc/




More information about the SciPy-user mailing list