[Numpy-discussion] tighten up ufunc casting rule

Ralf Gommers ralf.gommers@googlemail....
Sun Jun 5 15:43:45 CDT 2011


On Thu, Jun 2, 2011 at 10:12 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:

> On Thu, Jun 2, 2011 at 3:09 PM, Gael Varoquaux <
> gael.varoquaux@normalesup.org> wrote:
>
>> On Thu, Jun 02, 2011 at 03:06:58PM -0500, Mark Wiebe wrote:
>> >    Would anyone object to, at least temporarily, tightening up the
>> default
>> >    ufunc casting rule to 'same_kind' in NumPy master? It's a one line
>> change,
>> >    so would be easy to undo, but such a change is very desirable in my
>> >    opinion.
>> >    This would raise an exception, since it's np.add(a, 1.9, out=a),
>> >    converting a float to an int:
>>
>> >    >>> a = np.arange(3, dtype=np.int32)
>>
>> >    >>> a += 1.9
>>
>> That's probably going to break a huge amount of code which relies on the
>> current behavior.
>>
>> Am I right in believing that this should only be considered for a major
>> release of numpy, say numpy 2.0?
>
>
> Absolutely, and that's why I'm proposing to do it in master now, fairly
> early in a development cycle, so we can evaluate its effects. If the next
> version is 1.7, we probably would roll it back for release (a 1 line
> change), and if the next version is 2.0, we probably would keep it in.
>
> I suspect at least some of the code relying on the current behavior may
> have bugs, and tightening this up is a way to reveal them.
>
>
Here are some results of testing your tighten_casting branch on a few
projects - no need to first put it in master first to do that. Four failures
in numpy, two in scipy, four in scikit-learn (plus two that don't look
related), none in scikits.statsmodels. I didn't check how many of them are
actual bugs.

I'm not against trying out your change, but it would probably be good to do
some more testing first and fix the issues found before putting it in. Then
at least if people run into issues with the already tested packages, you can
just tell them to update those to latest master.

Cheers,
Ralf


NUMPY

======================================================================
ERROR: test_ones_like (test_numeric.TestLikeFuncs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rgommers/Code/numpy/numpy/core/tests/test_numeric.py", line
1257, in test_ones_like
    self.check_like_function(np.ones_like, 1)
  File "/Users/rgommers/Code/numpy/numpy/core/tests/test_numeric.py", line
1200, in check_like_function
    dz = like_function(d, dtype=dtype)
TypeError: found a loop for ufunc 'ones_like' matching the type-tuple, but
the inputs and/or outputs could not be cast according to the casting rule

======================================================================
ERROR: test_methods_with_output (test_core.TestMaskedArrayArithmetic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rgommers/Code/numpy/numpy/ma/tests/test_core.py", line 1109,
in test_methods_with_output
    result = xmmeth(axis=0, out=output)
  File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 4774, in std
    out **= 0.5
  File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 3765, in __ipow__
    ndarray.__ipow__(self._data, np.where(self._mask, 1, other_data))
TypeError: ufunc 'power' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
'same_kind'

======================================================================
ERROR: Check that we don't shrink a mask when not wanted
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rgommers/Code/numpy/numpy/ma/tests/test_core.py", line 1044,
in test_noshrinking
    a /= 1.
  File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 3725, in __idiv__
    ndarray.__idiv__(self._data, np.where(self._mask, 1, other_data))
TypeError: ufunc 'divide' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
'same_kind'

======================================================================
ERROR: Test of inplace additions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rgommers/Code/numpy/numpy/ma/tests/test_core.py", line 1651,
in test_inplace_addition_array
    x += a
  File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 3686, in __iadd__
    ndarray.__iadd__(self._data, np.where(self._mask, 0, getdata(other)))
TypeError: ufunc 'add' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
'same_kind'



SCIPY

======================================================================
ERROR: gaussian gradient magnitude filter 1
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
line 187, in runTest
    self.test(*self.arg)
  File "/Users/rgommers/Code/scipy/scipy/ndimage/tests/test_ndimage.py",
line 670, in test_gaussian_gradient_magnitude01
    1.0)
  File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 479, in
gaussian_gradient_magnitude
    cval, extra_arguments = (sigma,))
  File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 450, in
generic_gradient_magnitude
    numpy.sqrt(output, output)
TypeError: ufunc 'sqrt' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
'same_kind'

======================================================================
ERROR: gaussian gradient magnitude filter 2
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
line 187, in runTest
    self.test(*self.arg)
  File "/Users/rgommers/Code/scipy/scipy/ndimage/tests/test_ndimage.py",
line 685, in test_gaussian_gradient_magnitude02
    output)
  File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 479, in
gaussian_gradient_magnitude
    cval, extra_arguments = (sigma,))
  File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 450, in
generic_gradient_magnitude
    numpy.sqrt(output, output)
TypeError: ufunc 'sqrt' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'l') according to the casting rule
'same_kind'



SCIKIT-LEARN

======================================================================
ERROR: Test decision_function
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
line 187, in runTest
    self.test(*self.arg)
  File
"/Users/rgommers/Code/scikit-learn/scikits/learn/svm/tests/test_svm.py",
line 254, in test_decision_function
    assert_array_almost_equal(-dec, np.ravel(clf.decision_function(data)))
  File "/Users/rgommers/Code/scikit-learn/scikits/learn/svm/base.py", line
276, in decision_function
    **self._get_params())
  File "libsvm.pyx", line 385, in scikits.learn.svm.libsvm.decision_function
(scikits/learn/svm/libsvm.c:4748)
ValueError: ndarray is not C-contiguous

======================================================================
ERROR: Test weights on individual samples
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
line 187, in runTest
    self.test(*self.arg)
  File
"/Users/rgommers/Code/scikit-learn/scikits/learn/svm/tests/test_svm.py",
line 282, in test_sample_weights
    assert_array_equal(clf.predict(X[2]), [1.])
  File "/Users/rgommers/Code/scikit-learn/scikits/learn/svm/base.py", line
186, in predict
    svm_type=svm_type, **self._get_params())
  File "libsvm.pyx", line 224, in scikits.learn.svm.libsvm.predict
(scikits/learn/svm/libsvm.c:3133)
ValueError: ndarray is not C-contiguous

======================================================================
FAIL: Doctest: scikits.learn.decomposition.pca.PCA
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 2145, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for scikits.learn.decomposition.pca.PCA
  File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
94, in PCA

----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py",
line 161, in scikits.learn.decomposition.pca.PCA
Failed example:
    pca.fit(X)
Exception raised:
    Traceback (most recent call last):
      File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest scikits.learn.decomposition.pca.PCA[4]>", line 1, in
<module>
        pca.fit(X)
      File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
191, in fit
        self._fit(X, **params)
      File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
227, in _fit
        X -= self.mean_
    TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
to provided output parameter (typecode 'l') according to the casting rule
'same_kind'
----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py",
line 163, in scikits.learn.decomposition.pca.PCA
Failed example:
    print pca.explained_variance_ratio_
Exception raised:
    Traceback (most recent call last):
      File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest scikits.learn.decomposition.pca.PCA[5]>", line 1, in
<module>
        print pca.explained_variance_ratio_
    AttributeError: 'PCA' object has no attribute
'explained_variance_ratio_'

>>  raise self.failureException(self.format_failure(<StringIO.StringIO
instance at 0xfe05648>.getvalue()))


======================================================================
FAIL: Doctest: scikits.learn.decomposition.pca.ProbabilisticPCA
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 2145, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
scikits.learn.decomposition.pca.ProbabilisticPCA
  File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
273, in ProbabilisticPCA

----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py",
line 342, in scikits.learn.decomposition.pca.ProbabilisticPCA
Failed example:
    pca.fit(X)
Exception raised:
    Traceback (most recent call last):
      File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest scikits.learn.decomposition.pca.ProbabilisticPCA[4]>",
line 1, in <module>
        pca.fit(X)
      File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
191, in fit
        self._fit(X, **params)
      File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
227, in _fit
        X -= self.mean_
    TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
to provided output parameter (typecode 'l') according to the casting rule
'same_kind'
----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py",
line 344, in scikits.learn.decomposition.pca.ProbabilisticPCA
Failed example:
    print pca.explained_variance_ratio_
Exception raised:
    Traceback (most recent call last):
      File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest scikits.learn.decomposition.pca.ProbabilisticPCA[5]>",
line 1, in <module>
        print pca.explained_variance_ratio_
    AttributeError: 'PCA' object has no attribute
'explained_variance_ratio_'

>>  raise self.failureException(self.format_failure(<StringIO.StringIO
instance at 0xfd8d1c0>.getvalue()))


======================================================================
FAIL: Doctest: scikits.learn.decomposition.pca.RandomizedPCA
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 2145, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
scikits.learn.decomposition.pca.RandomizedPCA
  File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
330, in RandomizedPCA

----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py",
line 377, in scikits.learn.decomposition.pca.RandomizedPCA
Failed example:
    pca.fit(X)
Exception raised:
    Traceback (most recent call last):
      File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest scikits.learn.decomposition.pca.RandomizedPCA[4]>",
line 1, in <module>
        pca.fit(X)
      File
"/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
438, in fit
        X -= self.mean_
    TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
to provided output parameter (typecode 'l') according to the casting rule
'same_kind'
----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py",
line 379, in scikits.learn.decomposition.pca.RandomizedPCA
Failed example:
    print pca.explained_variance_ratio_
Exception raised:
    Traceback (most recent call last):
      File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest scikits.learn.decomposition.pca.RandomizedPCA[5]>",
line 1, in <module>
        print pca.explained_variance_ratio_
    AttributeError: 'RandomizedPCA' object has no attribute
'explained_variance_ratio_'

>>  raise self.failureException(self.format_failure(<StringIO.StringIO
instance at 0xfe05670>.getvalue()))


======================================================================
FAIL: Doctest: scikits.learn.mixture.GMM
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 2145, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for scikits.learn.mixture.GMM
  File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py", line
127, in GMM

----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py", line 226,
in scikits.learn.mixture.GMM
Failed example:
    g.fit(20 * [[0]] +  20 * [[10]])
Exception raised:
    Traceback (most recent call last):
      File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest scikits.learn.mixture.GMM[14]>", line 1, in <module>
        g.fit(20 * [[0]] +  20 * [[10]])
      File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py",
line 480, in fit
        k=self._n_states).fit(X).cluster_centers_
      File
"/Users/rgommers/Code/scikit-learn/scikits/learn/cluster/k_means_.py", line
432, in fit
        tol=self.tol, random_state=self.random_state, copy_x=self.copy_x)
      File
"/Users/rgommers/Code/scikit-learn/scikits/learn/cluster/k_means_.py", line
199, in k_means
        X -= Xmean
    TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
to provided output parameter (typecode 'l') according to the casting rule
'same_kind'
----------------------------------------------------------------------
File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py", line 228,
in scikits.learn.mixture.GMM
Failed example:
    np.round(g.weights, 2)
Expected:
    array([ 0.5,  0.5])
Got:
    array([ 0.25,  0.75])

>>  raise self.failureException(self.format_failure(<StringIO.StringIO
instance at 0xfde5238>.getvalue()))
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110605/c928c313/attachment-0001.html 


More information about the NumPy-Discussion mailing list