[Numpy-discussion] tighten up ufunc casting rule

Mark Wiebe mwwiebe@gmail....
Mon Jun 6 10:30:01 CDT 2011


On Sun, Jun 5, 2011 at 3:43 PM, Ralf Gommers <ralf.gommers@googlemail.com>wrote:

> On Thu, Jun 2, 2011 at 10:12 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
>
>> On Thu, Jun 2, 2011 at 3:09 PM, Gael Varoquaux <
>> gael.varoquaux@normalesup.org> wrote:
>>
>>> On Thu, Jun 02, 2011 at 03:06:58PM -0500, Mark Wiebe wrote:
>>> >    Would anyone object to, at least temporarily, tightening up the
>>> default
>>> >    ufunc casting rule to 'same_kind' in NumPy master? It's a one line
>>> change,
>>> >    so would be easy to undo, but such a change is very desirable in my
>>> >    opinion.
>>> >    This would raise an exception, since it's np.add(a, 1.9, out=a),
>>> >    converting a float to an int:
>>>
>>> >    >>> a = np.arange(3, dtype=np.int32)
>>>
>>> >    >>> a += 1.9
>>>
>>> That's probably going to break a huge amount of code which relies on the
>>> current behavior.
>>>
>>> Am I right in believing that this should only be considered for a major
>>> release of numpy, say numpy 2.0?
>>
>>
>> Absolutely, and that's why I'm proposing to do it in master now, fairly
>> early in a development cycle, so we can evaluate its effects. If the next
>> version is 1.7, we probably would roll it back for release (a 1 line
>> change), and if the next version is 2.0, we probably would keep it in.
>>
>> I suspect at least some of the code relying on the current behavior may
>> have bugs, and tightening this up is a way to reveal them.
>>
>>
> Here are some results of testing your tighten_casting branch on a few
> projects - no need to first put it in master first to do that. Four failures
> in numpy, two in scipy, four in scikit-learn (plus two that don't look
> related), none in scikits.statsmodels. I didn't check how many of them are
> actual bugs.
>
> I'm not against trying out your change, but it would probably be good to do
> some more testing first and fix the issues found before putting it in. Then
> at least if people run into issues with the already tested packages, you can
> just tell them to update those to latest master.
>

Cool, thanks for running those. I already took a chunk out of the NumPy
failures. The ones_like function shouldn't really be a ufunc, but rather be
like zeros_like and empty_like, but that's probably not something to change
right now. The datetime-fixes type resolution change provides a mechanism to
fix that up pretty easily.

For Scipy, what do you think is the best way to resolve it? If NumPy 1.6 is
the minimum version for the next scipy, I would add casting='unsafe' to the
failing sqrt call.

-Mark


> Cheers,
> Ralf
>
>
> NUMPY
>
> ======================================================================
> ERROR: test_ones_like (test_numeric.TestLikeFuncs)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/rgommers/Code/numpy/numpy/core/tests/test_numeric.py", line
> 1257, in test_ones_like
>     self.check_like_function(np.ones_like, 1)
>   File "/Users/rgommers/Code/numpy/numpy/core/tests/test_numeric.py", line
> 1200, in check_like_function
>     dz = like_function(d, dtype=dtype)
> TypeError: found a loop for ufunc 'ones_like' matching the type-tuple, but
> the inputs and/or outputs could not be cast according to the casting rule
>
> ======================================================================
> ERROR: test_methods_with_output (test_core.TestMaskedArrayArithmetic)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/rgommers/Code/numpy/numpy/ma/tests/test_core.py", line 1109,
> in test_methods_with_output
>     result = xmmeth(axis=0, out=output)
>   File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 4774, in std
>     out **= 0.5
>   File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 3765, in
> __ipow__
>     ndarray.__ipow__(self._data, np.where(self._mask, 1, other_data))
> TypeError: ufunc 'power' output (typecode 'd') could not be coerced to
> provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
>
> ======================================================================
> ERROR: Check that we don't shrink a mask when not wanted
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/rgommers/Code/numpy/numpy/ma/tests/test_core.py", line 1044,
> in test_noshrinking
>     a /= 1.
>   File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 3725, in
> __idiv__
>     ndarray.__idiv__(self._data, np.where(self._mask, 1, other_data))
> TypeError: ufunc 'divide' output (typecode 'd') could not be coerced to
> provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
>
> ======================================================================
> ERROR: Test of inplace additions
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/rgommers/Code/numpy/numpy/ma/tests/test_core.py", line 1651,
> in test_inplace_addition_array
>     x += a
>   File "/Users/rgommers/Code/numpy/numpy/ma/core.py", line 3686, in
> __iadd__
>     ndarray.__iadd__(self._data, np.where(self._mask, 0, getdata(other)))
> TypeError: ufunc 'add' output (typecode 'd') could not be coerced to
> provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
>
>
>
> SCIPY
>
> ======================================================================
> ERROR: gaussian gradient magnitude filter 1
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
> line 187, in runTest
>     self.test(*self.arg)
>   File "/Users/rgommers/Code/scipy/scipy/ndimage/tests/test_ndimage.py",
> line 670, in test_gaussian_gradient_magnitude01
>     1.0)
>   File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 479, in
> gaussian_gradient_magnitude
>     cval, extra_arguments = (sigma,))
>   File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 450, in
> generic_gradient_magnitude
>     numpy.sqrt(output, output)
> TypeError: ufunc 'sqrt' output (typecode 'd') could not be coerced to
> provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
>
> ======================================================================
> ERROR: gaussian gradient magnitude filter 2
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
> line 187, in runTest
>     self.test(*self.arg)
>   File "/Users/rgommers/Code/scipy/scipy/ndimage/tests/test_ndimage.py",
> line 685, in test_gaussian_gradient_magnitude02
>     output)
>   File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 479, in
> gaussian_gradient_magnitude
>     cval, extra_arguments = (sigma,))
>   File "/Users/rgommers/Code/scipy/scipy/ndimage/filters.py", line 450, in
> generic_gradient_magnitude
>     numpy.sqrt(output, output)
> TypeError: ufunc 'sqrt' output (typecode 'd') could not be coerced to
> provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
>
>
>
> SCIKIT-LEARN
>
> ======================================================================
> ERROR: Test decision_function
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
> line 187, in runTest
>     self.test(*self.arg)
>   File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/svm/tests/test_svm.py",
> line 254, in test_decision_function
>     assert_array_almost_equal(-dec, np.ravel(clf.decision_function(data)))
>   File "/Users/rgommers/Code/scikit-learn/scikits/learn/svm/base.py", line
> 276, in decision_function
>     **self._get_params())
>   File "libsvm.pyx", line 385, in
> scikits.learn.svm.libsvm.decision_function (scikits/learn/svm/libsvm.c:4748)
> ValueError: ndarray is not C-contiguous
>
> ======================================================================
> ERROR: Test weights on individual samples
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.0.0-py2.6.egg/nose/case.py",
> line 187, in runTest
>     self.test(*self.arg)
>   File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/svm/tests/test_svm.py",
> line 282, in test_sample_weights
>     assert_array_equal(clf.predict(X[2]), [1.])
>   File "/Users/rgommers/Code/scikit-learn/scikits/learn/svm/base.py", line
> 186, in predict
>     svm_type=svm_type, **self._get_params())
>   File "libsvm.pyx", line 224, in scikits.learn.svm.libsvm.predict
> (scikits/learn/svm/libsvm.c:3133)
> ValueError: ndarray is not C-contiguous
>
> ======================================================================
> FAIL: Doctest: scikits.learn.decomposition.pca.PCA
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 2145, in runTest
>     raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for scikits.learn.decomposition.pca.PCA
>   File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 94, in PCA
>
> ----------------------------------------------------------------------
> File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 161, in scikits.learn.decomposition.pca.PCA
> Failed example:
>     pca.fit(X)
> Exception raised:
>     Traceback (most recent call last):
>       File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1241, in __run
>         compileflags, 1) in test.globs
>       File "<doctest scikits.learn.decomposition.pca.PCA[4]>", line 1, in
> <module>
>         pca.fit(X)
>       File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 191, in fit
>         self._fit(X, **params)
>       File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 227, in _fit
>         X -= self.mean_
>     TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
> to provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
> ----------------------------------------------------------------------
> File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 163, in scikits.learn.decomposition.pca.PCA
> Failed example:
>     print pca.explained_variance_ratio_
> Exception raised:
>     Traceback (most recent call last):
>       File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1241, in __run
>         compileflags, 1) in test.globs
>       File "<doctest scikits.learn.decomposition.pca.PCA[5]>", line 1, in
> <module>
>         print pca.explained_variance_ratio_
>     AttributeError: 'PCA' object has no attribute
> 'explained_variance_ratio_'
>
> >>  raise self.failureException(self.format_failure(<StringIO.StringIO
> instance at 0xfe05648>.getvalue()))
>
>
> ======================================================================
> FAIL: Doctest: scikits.learn.decomposition.pca.ProbabilisticPCA
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 2145, in runTest
>     raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for
> scikits.learn.decomposition.pca.ProbabilisticPCA
>   File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 273, in ProbabilisticPCA
>
> ----------------------------------------------------------------------
> File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 342, in scikits.learn.decomposition.pca.ProbabilisticPCA
> Failed example:
>     pca.fit(X)
> Exception raised:
>     Traceback (most recent call last):
>       File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1241, in __run
>         compileflags, 1) in test.globs
>       File "<doctest scikits.learn.decomposition.pca.ProbabilisticPCA[4]>",
> line 1, in <module>
>         pca.fit(X)
>       File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 191, in fit
>         self._fit(X, **params)
>       File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 227, in _fit
>         X -= self.mean_
>     TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
> to provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
> ----------------------------------------------------------------------
> File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 344, in scikits.learn.decomposition.pca.ProbabilisticPCA
> Failed example:
>     print pca.explained_variance_ratio_
> Exception raised:
>     Traceback (most recent call last):
>       File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1241, in __run
>         compileflags, 1) in test.globs
>       File "<doctest scikits.learn.decomposition.pca.ProbabilisticPCA[5]>",
> line 1, in <module>
>         print pca.explained_variance_ratio_
>     AttributeError: 'PCA' object has no attribute
> 'explained_variance_ratio_'
>
> >>  raise self.failureException(self.format_failure(<StringIO.StringIO
> instance at 0xfd8d1c0>.getvalue()))
>
>
> ======================================================================
> FAIL: Doctest: scikits.learn.decomposition.pca.RandomizedPCA
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 2145, in runTest
>     raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for
> scikits.learn.decomposition.pca.RandomizedPCA
>   File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 330, in RandomizedPCA
>
> ----------------------------------------------------------------------
> File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 377, in scikits.learn.decomposition.pca.RandomizedPCA
> Failed example:
>     pca.fit(X)
> Exception raised:
>     Traceback (most recent call last):
>       File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1241, in __run
>         compileflags, 1) in test.globs
>       File "<doctest scikits.learn.decomposition.pca.RandomizedPCA[4]>",
> line 1, in <module>
>         pca.fit(X)
>       File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 438, in fit
>         X -= self.mean_
>     TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
> to provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
> ----------------------------------------------------------------------
> File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/decomposition/pca.py", line
> 379, in scikits.learn.decomposition.pca.RandomizedPCA
> Failed example:
>     print pca.explained_variance_ratio_
> Exception raised:
>     Traceback (most recent call last):
>       File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1241, in __run
>         compileflags, 1) in test.globs
>       File "<doctest scikits.learn.decomposition.pca.RandomizedPCA[5]>",
> line 1, in <module>
>         print pca.explained_variance_ratio_
>     AttributeError: 'RandomizedPCA' object has no attribute
> 'explained_variance_ratio_'
>
> >>  raise self.failureException(self.format_failure(<StringIO.StringIO
> instance at 0xfe05670>.getvalue()))
>
>
> ======================================================================
> FAIL: Doctest: scikits.learn.mixture.GMM
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 2145, in runTest
>     raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for scikits.learn.mixture.GMM
>   File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py", line
> 127, in GMM
>
> ----------------------------------------------------------------------
> File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py", line
> 226, in scikits.learn.mixture.GMM
> Failed example:
>     g.fit(20 * [[0]] +  20 * [[10]])
> Exception raised:
>     Traceback (most recent call last):
>       File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py",
> line 1241, in __run
>         compileflags, 1) in test.globs
>       File "<doctest scikits.learn.mixture.GMM[14]>", line 1, in <module>
>         g.fit(20 * [[0]] +  20 * [[10]])
>       File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py",
> line 480, in fit
>         k=self._n_states).fit(X).cluster_centers_
>       File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/cluster/k_means_.py", line
> 432, in fit
>         tol=self.tol, random_state=self.random_state, copy_x=self.copy_x)
>       File
> "/Users/rgommers/Code/scikit-learn/scikits/learn/cluster/k_means_.py", line
> 199, in k_means
>         X -= Xmean
>     TypeError: ufunc 'subtract' output (typecode 'd') could not be coerced
> to provided output parameter (typecode 'l') according to the casting rule
> 'same_kind'
> ----------------------------------------------------------------------
> File "/Users/rgommers/Code/scikit-learn/scikits/learn/mixture.py", line
> 228, in scikits.learn.mixture.GMM
> Failed example:
>     np.round(g.weights, 2)
> Expected:
>     array([ 0.5,  0.5])
> Got:
>     array([ 0.25,  0.75])
>
> >>  raise self.failureException(self.format_failure(<StringIO.StringIO
> instance at 0xfde5238>.getvalue()))
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110606/1f85338c/attachment-0001.html 


More information about the NumPy-Discussion mailing list