[SciPy-dev] Re: Fundamental scipy testing changes

Todd Miller jmiller at stsci.edu
Tue Dec 7 17:11:36 CST 2004


The discussion below relates to porting scipy to numarray, starting with
some small changes to scipy_test.testing.  The discussion began as
private e-mail but has crossed over to scipy-dev at Pearu's suggestion.

On Tue, 2004-12-07 at 16:16, Pearu Peterson wrote:
> Hi Todd,
> 
> On 7 Dec 2004, Todd Miller wrote:
> 
> > I took a closer look at how my scipy_base changes affect scipy.test()
> > and found a couple things in my scipy_base changes that needed fixing. 
> > Sorry I didn't think of this sooner.  These fixes are committed now on
> > the numarray branch in CVS.  I also honed in on what I think should be
> > the starting point for discussion: the changes I made to
> > scipy_test/testing.py.
> 
> Note that assert_equal, assert_almost_equal, assert_approx_equal
> were not meant to be used with array arguments (I didn't implement them
> but its obvious from reading the code). 

Thanks for pointing this out...  I noticed the array versions only
peripherally and didn't understand the distinction.

> For checking the equality of 
> array arguments assert_array_equal or assert_array_almost_equal should be
> used. If some scipy test suite uses assert_equal, etc with array
> arguments then I think this is a bug of this particular test suite,
> not of testing.py. So, using scipy_base.all in assert_equal, etc is not
> necessary (unless we want to drop assert_array_* functions).

Understood.  Are we agreed that it is appropriate to use all() in the
assert_array_* functions?  

> > In order to do the numarray scipy_base changes and pass
> > scipy_base.test(),  I modified scipy_test/testing.py in ways which I
> > assert are good.  Mostly, I used all() in contexts where an array result
> > was being used as a truth value.  In those contexts,  the array
> > __nonzero__() function is executed.  NumArray.__nonzero__() raises an
> > exception causing many tests to fail that should succeed.  Using all()
> > reduces the "truth array" to a single scalar value,  so
> > NumArray.__nonzero__() is never called.
> >
> > Where I think early agreement needs to happen is that my testing.py
> > changes are good even though they expose a handful of latent scipy bugs
> > or unit test bugs because Numeric's __nonzero__() has the meaning of
> > any() and not all().   So,  for say an equality test,  if any of the
> > values were equal,  the test would succeed even if most of the arrays'
> > values were *not* equal.  Switching to all() as I am advocating exposes
> > hidden problems like these.  
> 
> At the moment I have no comments on this as I haven't tried the numarray
> branch of scipy yet.

I think we should square away the scipy_test.testing changes before
anyone messes with the numarray branch.

>  Btw, your discussion about the changes to scipy 
> sounds reasonable. However, why do you say that using alltrue is an error?
> I thought all(x) is equivalent to alltrue(ravel(x)) and in assert_array_*
> functions the arguments to alltrue are already ravel'ed.

I looked at my diffs more carefully and see that you're right, the
alltrues are not bugs because they're only fed 1D arrays.  I have not
reexamined all my scipy_base changes for instances of alltrue.

> > If my testing.py changes are agreed upon and the exposed bugs are either
> > fixed or acknowledged as known,  we have a better basis for examining
> > the rest of the numarray changes to scipy_base.
> > 
> > I attached 3 files for you to look at:
> > 
> > 1. testing.diffs  (changes to -r HEAD of scipy_test/testing.py)
> > 
> > 2. results.HEAD   (scipy.test() results against the HEAD of CVS using
> > Numeric)  
> > I got 4 failures and can't see the nightly scoreboard yet so I don't
> > know if these are expected.
> > 
> > 3. results.testing.changes  (scipy.test() results with testing.diffs
> > applied)
> > I got 11 failures (including the original 4) most of which I believe are
> > either bugs or unit test bugs.
> 
> The failures, where you see almost equal arrays, may occur when using
> Fortran blas; 

My scipy.test() does complain about not finding clapack so maybe I am
using a Fortran blas too.

> these errors should be fixed by using proper decimal
> argument in assert_array_almost_equal call of the corresponding tests (I
> thought I have fixed these errors in scipy HEAD already). 

As you said I think,  the (incorrect) changes to assert_equal exposed
(incorrect) uses of assert_equal in array contexts.

> Other failures need more attention.
> 
> > If any of you would rather not be included in my future numarray
> > mailings,  or you have suggestions for a better venue for them,  please
> > let me know.
> 
> I suggest using scipy-dev.
> 
> Thanks,
> Pearu

I re-attached the attachments in case anyone on scipy-dev wants to look
as well;  there's nothing new there.

Thanks for looking it over,
Todd

-------------- next part --------------
A non-text attachment was scrubbed...
Name: testing.diffs
Type: text/x-patch
Size: 5846 bytes
Desc: not available
Url : http://projects.scipy.org/pipermail/scipy-dev/attachments/20041207/a3842097/attachment.bin 
-------------- next part --------------
======================================================================
FAIL: check_heevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 80, in check_heevr_irange_high
    def check_heevr_irange_high(self): self.check_syevr_irange(sym='he',irange=[1,2])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269+0.j  4.0951085+0.j  7.3420534+0.j]
        Array 2: [ 3.6929276+0.j  4.0951101+0.j  7.3420528+0.j]


======================================================================
FAIL: check_heevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 102, in check_heevr_vrange
    def check_heevr_vrange(self): self.check_syevr_vrange(sym='he')
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269+0.j  4.0951085+0.j  7.3420534+0.j]
        Array 2: [ 3.6929276+0.j  4.0951101+0.j  7.3420528+0.j]


======================================================================
FAIL: check_syevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 74, in check_syevr_irange_high
    def check_syevr_irange_high(self): self.check_syevr_irange(irange=[1,2])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269  4.0951085  7.3420534]
        Array 2: [ 3.6929276  4.0951101  7.3420528]


======================================================================
FAIL: check_syevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269  4.0951085  7.3420534]
        Array 2: [ 3.6929276  4.0951101  7.3420528]


----------------------------------------------------------------------
Ran 1172 tests in 7.154s

FAILED (failures=4)
<unittest.TextTestRunner object at 0x4189b3b4>
[413475 refs]
-------------- next part --------------
======================================================================
FAIL: check_basic (scipy_base.function_base.test_function_base.test_amax)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_base/tests/test_function_base.py", line 72, in check_basic
    assert_equal(amax(b),[8.0,10.0,9.0])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
    assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: [8.0, 10.0, 9.0]
ACTUAL: array([  9.,  10.,   8.])

======================================================================
FAIL: check_basic (scipy_base.function_base.test_function_base.test_amin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_base/tests/test_function_base.py", line 82, in check_basic
    assert_equal(amin(b),[3.0,3.0,2.0])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
    assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: [3.0, 3.0, 2.0]
ACTUAL: array([ 3.,  4.,  2.])

======================================================================
FAIL: check_heevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 80, in check_heevr_irange_high
    def check_heevr_irange_high(self): self.check_syevr_irange(sym='he',irange=[1,2])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269+0.j  4.0951085+0.j  7.3420534+0.j]
        Array 2: [ 3.6929276+0.j  4.0951101+0.j  7.3420528+0.j]


======================================================================
FAIL: check_heevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 102, in check_heevr_vrange
    def check_heevr_vrange(self): self.check_syevr_vrange(sym='he')
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269+0.j  4.0951085+0.j  7.3420534+0.j]
        Array 2: [ 3.6929276+0.j  4.0951101+0.j  7.3420528+0.j]


======================================================================
FAIL: check_syevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 74, in check_syevr_irange_high
    def check_syevr_irange_high(self): self.check_syevr_irange(irange=[1,2])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269  4.0951085  7.3420534]
        Array 2: [ 3.6929276  4.0951101  7.3420528]


======================================================================
FAIL: check_syevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
    assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
    assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
        Array 1: [ 3.6929269  4.0951085  7.3420534]
        Array 2: [ 3.6929276  4.0951101  7.3420528]


======================================================================
FAIL: check_arange (scipy.special.basic.test_basic.test_arange)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/special/tests/test_basic.py", line 495, in check_arange
    assert_equal(numstring,array([0.,0.1,0.2,0.3,
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
    assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:


======================================================================
FAIL: check_genlaguerre (scipy.special.basic.test_basic.test_laguerre)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/special/tests/test_basic.py", line 1573, in check_genlaguerre
    assert_equal(lag2.c,array([1,-2*(k+2),(k+1.)*(k+2.)])/2.0)
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
    assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: array([ 0.5       , -3.58365527,  4.62946491])
ACTUAL: array([ 0.5       , -3.58365527,  4.62946491])

======================================================================
FAIL: check_legendre (scipy.special.basic.test_basic.test_legendre)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/special/tests/test_basic.py", line 1590, in check_legendre
    assert_equal(leg3.c,array([5,0,-3,0])/2.0)
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
    assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: array([ 2.5,  0. , -1.5,  0. ])
ACTUAL: array([  2.50000000e+00,   0.00000000e+00,  -1.50000000e+00,   3.10191936e-17])

======================================================================
FAIL: check_diag (scipy.linalg.basic.test_basic.test_tri)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/linalg/tests/test_basic.py", line 427, in check_diag
    assert_equal(tri(4,k=1),array([[1,1,0,0],
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
    assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: array([[1, 1, 0, 0],
       [1, 1, 1, 0],
       [0, 1, 1, 1],
       [1, 1, 1, 1]])
ACTUAL: array([[1, 1, 0, 0],
       [1, 1, 1, 0],
       [1, 1, 1, 1],
       [1, 1, 1, 1]],'b')

======================================================================
FAIL: check_diag (scipy.linalg.basic.test_basic.test_triu)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy/linalg/tests/test_basic.py", line 494, in check_diag
    assert_equal(tril(a,k=-2),b)
  File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
    assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:


----------------------------------------------------------------------
Ran 1172 tests in 6.999s

FAILED (failures=11)
<unittest.TextTestRunner object at 0x4189c424>
[455000 refs]
>>>


More information about the Scipy-dev mailing list