[Numpy-discussion] NA masks for NumPy are ready to test

Bruce Southey bsouthey@gmail....
Fri Aug 19 09:55:46 CDT 2011


On 08/18/2011 04:43 PM, Mark Wiebe wrote:
> It's taken a lot of changes to get the NA mask support to its current 
> point, but the code ready for some testing now. You can read the 
> work-in-progress release notes here:
>
> https://github.com/m-paradox/numpy/blob/missingdata/doc/release/2.0.0-notes.rst
>
> To try it out, check out the missingdata branch from my github 
> account, here, and build in the standard way:
>
> https://github.com/m-paradox/numpy
>
> The things most important to test are:
>
> * Confirm that existing code still works correctly. I've tested 
> against SciPy and matplotlib.
> * Confirm that the performance of code not using NA masks is the same 
> or better.
> * Try to do computations with the NA values, find places they don't 
> work yet, and nominate unimplemented functionality important to you to 
> be next on the development list. The release notes have a preliminary 
> list of implemented/unimplemented functions.
> * Report any crashes, build problems, or unexpected behaviors.
>
> In addition to adding the NA mask, I've also added features and done a 
> few performance changes here and there, like letting reductions like 
> sum take lists of axes instead of being a single axis or all of them. 
> These changes affect various bugs like 
> http://projects.scipy.org/numpy/ticket/1143 and 
> http://projects.scipy.org/numpy/ticket/533.
>
> Thanks!
> Mark
>
> Here's a small example run using NAs:
>
> >>> import numpy as np
> >>> np.__version__
> '2.0.0.dev-8a5e2a1'
> >>> a = np.random.rand(3,3,3)
> >>> a.flags.maskna = True
> >>> a[np.random.rand(3,3,3) < 0.5] = np.NA
> >>> a
> array([[[NA, NA,  0.11511708],
>         [ 0.46661454,  0.47565512, NA],
>         [NA, NA, NA]],
>
>        [[NA,  0.57860351, NA],
>         [NA, NA,  0.72012669],
>         [ 0.36582123, NA,  0.76289794]],
>
>        [[ 0.65322748,  0.92794386, NA],
>         [ 0.53745165,  0.97520989,  0.17515083],
>         [ 0.71219688,  0.5184328 ,  0.75802805]]])
> >>> np.mean(a, axis=-1)
> array([[NA, NA, NA],
>        [NA, NA, NA],
>        [NA,  0.56260412,  0.66288591]])
> >>> np.std(a, axis=-1)
> array([[NA, NA, NA],
>        [NA, NA, NA],
>        [NA,  0.32710662,  0.10384331]])
> >>> np.mean(a, axis=-1, skipna=True)
> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2474: 
> RuntimeWarning: invalid value encountered in true_divide
>   um.true_divide(ret, rcount, out=ret, casting='unsafe')
> array([[ 0.11511708,  0.47113483,         nan],
>        [ 0.57860351,  0.72012669,  0.56435958],
>        [ 0.79058567,  0.56260412,  0.66288591]])
> >>> np.std(a, axis=-1, skipna=True)
> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2707: 
> RuntimeWarning: invalid value encountered in true_divide
>   um.true_divide(arrmean, rcount, out=arrmean, casting='unsafe')
> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2730: 
> RuntimeWarning: invalid value encountered in true_divide
>   um.true_divide(ret, rcount, out=ret, casting='unsafe')
> array([[ 0.        ,  0.00452029,         nan],
>        [ 0.        ,  0.        ,  0.19853835],
>        [ 0.13735819,  0.32710662,  0.10384331]])
> >>> np.std(a, axis=(1,2), skipna=True)
> array([ 0.16786895,  0.15498008,  0.23811937])
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
I had to rebuild my Python2.6 as a 'normal' version.

Anyhow, Python2.4, 2.5, 2.6 and 2.7 all build and pass the numpy tests.

Curiously, only tests in Python2.7 give almost no warnings but all the 
other Python2.x give lots of warnings - Python2.6 and Python2.7 are 
below. My expectation is that all versions should behave the same 
regarding printing messages.

Also the message 'Need pytz library to test datetime timezones' means 
that there are invalid tests that have to be rewritten (ticket 1939:  
http://projects.scipy.org/numpy/ticket/1939 ).

Bruce

$ python2.6 -c "import numpy; numpy.test()"
Running unit tests for numpy
NumPy version 2.0.0.dev-93236a2
NumPy is installed in /usr/local/lib/python2.6/site-packages/numpy
Python version 2.6.6 (r266:84292, Aug 19 2011, 09:21:38) [GCC 4.5.1 
20100924 (Red Hat 4.5.1-4)]
nose version 1.0.0
......................../usr/local/lib/python2.6/site-packages/numpy/core/tests/test_datetime.py:1313: 
UserWarning: Need pytz library to test datetime timezones
   warnings.warn("Need pytz library to test datetime timezones")
.........................................................................................................................../usr/local/lib/python2.6/unittest.py:336: 
DeprecationWarning: DType strings 'O4' and 'O8' are deprecated because 
they are platform specific. Use 'O' instead
   callableObj(*args, **kwargs)
............................................................................................................................................................................................................./usr/local/lib/python2.6/site-packages/numpy/core/_internal.py:555: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   value.names = tuple(names)
...../usr/local/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py:1912: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   dt.names = tuple(names)
...../usr/local/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py:804: 
DeprecationWarning: DType strings 'O4' and 'O8' are deprecated because 
they are platform specific. Use 'O' instead
   return loads(obj)
..../usr/local/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py:1046: 
DeprecationWarning: putmask has been deprecated. Use copyto with 'where' 
as the mask instead
   np.putmask(x,[True,False,True],-1)
../usr/local/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py:1025: 
DeprecationWarning: putmask has been deprecated. Use copyto with 'where' 
as the mask instead
   np.putmask(x, mask, val)
................................................/usr/local/lib/python2.6/unittest.py:336: 
DeprecationWarning: putmask has been deprecated. Use copyto with 'where' 
as the mask instead
   callableObj(*args, **kwargs)
../usr/local/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py:1057: 
DeprecationWarning: putmask has been deprecated. Use copyto with 'where' 
as the mask instead
   np.putmask(rec['x'],[True,False],10)
/usr/local/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py:1061: 
DeprecationWarning: putmask has been deprecated. Use copyto with 'where' 
as the mask instead
   np.putmask(rec['y'],[True,False],11)
.S/usr/local/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py:1395: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   dt.names = ['p','q']
..................................................................................................................................................................................................................................................................................................................................................................................../usr/local/lib/python2.6/site-packages/numpy/core/records.py:157: 
DeprecationWarning: DType strings 'O4' and 'O8' are deprecated because 
they are platform specific. Use 'O' instead
   dtype = sb.dtype(formats, aligned)
........................................................./usr/local/lib/python2.6/site-packages/numpy/core/tests/test_regression.py:1426: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   ra.dtype.names = ('f1', 'f2')
/usr/local/lib/python2.6/unittest.py:336: DeprecationWarning: Setting 
NumPy dtype names is deprecated, the dtype will become immutable in a 
future version
   callableObj(*args, **kwargs)
............../usr/local/lib/python2.6/site-packages/numpy/core/tests/test_regression.py:1017: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   a.dtype.names = b
......................................................................................................................./usr/local/lib/python2.6/pickle.py:1133: 
DeprecationWarning: DType strings 'O4' and 'O8' are deprecated because 
they are platform specific. Use 'O' instead
   value = func(*args)
..........................................................................................K..................................................................................................K......................K..........................................................................................................S...................................../usr/local/lib/python2.6/site-packages/numpy/lib/_iotools.py:857: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   ndtype.names = validate(ndtype.names, defaultfmt=defaultfmt)
/usr/local/lib/python2.6/site-packages/numpy/lib/_iotools.py:854: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   ndtype.names = validate([''] * nbtypes, defaultfmt=defaultfmt)
/usr/local/lib/python2.6/site-packages/numpy/lib/_iotools.py:847: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   defaultfmt=defaultfmt)
......................................................................................................................................................................................./usr/local/lib/python2.6/site-packages/numpy/lib/format.py:358: 
DeprecationWarning: DType strings 'O4' and 'O8' are deprecated because 
they are platform specific. Use 'O' instead
   dtype = numpy.dtype(d['descr'])
/usr/local/lib/python2.6/site-packages/numpy/lib/format.py:449: 
DeprecationWarning: DType strings 'O4' and 'O8' are deprecated because 
they are platform specific. Use 'O' instead
   array = cPickle.load(fp)
.............................................................................................................................................................................................................................................................................................................................../usr/local/lib/python2.6/site-packages/numpy/ma/core.py:366: 
DeprecationWarning: DType strings 'O4' and 'O8' are deprecated because 
they are platform specific. Use 'O' instead
   deflist.append(default_fill_value(np.dtype(currenttype)))
................/usr/local/lib/python2.6/site-packages/numpy/lib/npyio.py:1640: 
DeprecationWarning: Setting NumPy dtype names is deprecated, the dtype 
will become immutable in a future version
   dtype.names = names
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 3064 tests in 22.795s

OK (KNOWNFAIL=3, SKIP=2)
$ python -c "import numpy; numpy.test()"
Running unit tests for numpy
NumPy version 2.0.0.dev-93236a2
NumPy is installed in /usr/lib64/python2.7/site-packages/numpy
Python version 2.7 (r27:82500, Sep 16 2010, 18:02:00) [GCC 4.5.1 
20100907 (Red Hat 4.5.1-3)]
nose version 1.0.0
......................../usr/lib64/python2.7/site-packages/numpy/core/tests/test_datetime.py:1313: 
UserWarning: Need pytz library to test datetime timezones
   warnings.warn("Need pytz library to test datetime timezones")
...........................................................................................................................................................................................................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K..................................................................................................K......................K..........................................................................................................S...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 3064 tests in 23.180s

OK (KNOWNFAIL=3, SKIP=2)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20110819/7b6322b4/attachment-0001.html 


More information about the NumPy-Discussion mailing list