<br><br><div class="gmail_quote">On Fri, Aug 19, 2011 at 11:55 AM, Bruce Southey <span dir="ltr">&lt;<a href="mailto:bsouthey@gmail.com">bsouthey@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5">On Fri, Aug 19, 2011 at 10:48 AM, Mark Wiebe &lt;<a href="mailto:mwwiebe@gmail.com">mwwiebe@gmail.com</a>&gt; wrote:<br>
&gt; On Fri, Aug 19, 2011 at 7:15 AM, Bruce Southey &lt;<a href="mailto:bsouthey@gmail.com">bsouthey@gmail.com</a>&gt; wrote:<br>
&gt;&gt;<br>
&gt;&gt; On 08/18/2011 04:43 PM, Mark Wiebe wrote:<br>
&gt;&gt;<br>
&gt;&gt; It&#39;s taken a lot of changes to get the NA mask support to its current<br>
&gt;&gt; point, but the code ready for some testing now. You can read the<br>
&gt;&gt; work-in-progress release notes here:<br>
&gt;&gt;<br>
&gt;&gt; <a href="https://github.com/m-paradox/numpy/blob/missingdata/doc/release/2.0.0-notes.rst" target="_blank">https://github.com/m-paradox/numpy/blob/missingdata/doc/release/2.0.0-notes.rst</a><br>
&gt;&gt; To try it out, check out the missingdata branch from my github account,<br>
&gt;&gt; here, and build in the standard way:<br>
&gt;&gt; <a href="https://github.com/m-paradox/numpy" target="_blank">https://github.com/m-paradox/numpy</a><br>
&gt;&gt; The things most important to test are:<br>
&gt;&gt; * Confirm that existing code still works correctly. I&#39;ve tested against<br>
&gt;&gt; SciPy and matplotlib.<br>
&gt;&gt; * Confirm that the performance of code not using NA masks is the same or<br>
&gt;&gt; better.<br>
&gt;&gt; * Try to do computations with the NA values, find places they don&#39;t work<br>
&gt;&gt; yet, and nominate unimplemented functionality important to you to be next on<br>
&gt;&gt; the development list. The release notes have a preliminary list of<br>
&gt;&gt; implemented/unimplemented functions.<br>
&gt;&gt; * Report any crashes, build problems, or unexpected behaviors.<br>
&gt;&gt; In addition to adding the NA mask, I&#39;ve also added features and done a few<br>
&gt;&gt; performance changes here and there, like letting reductions like sum take<br>
&gt;&gt; lists of axes instead of being a single axis or all of them. These changes<br>
&gt;&gt; affect various bugs<br>
&gt;&gt; like†<a href="http://projects.scipy.org/numpy/ticket/1143" target="_blank">http://projects.scipy.org/numpy/ticket/1143</a>†and†<a href="http://projects.scipy.org/numpy/ticket/533" target="_blank">http://projects.scipy.org/numpy/ticket/533</a>.<br>

&gt;&gt; Thanks!<br>
&gt;&gt; Mark<br>
&gt;&gt; Here&#39;s a small example run using NAs:<br>
&gt;&gt; &gt;&gt;&gt; import numpy as np<br>
&gt;&gt; &gt;&gt;&gt; np.__version__<br>
&gt;&gt; &#39;2.0.0.dev-8a5e2a1&#39;<br>
&gt;&gt; &gt;&gt;&gt; a = np.random.rand(3,3,3)<br>
&gt;&gt; &gt;&gt;&gt; a.flags.maskna = True<br>
&gt;&gt; &gt;&gt;&gt; a[np.random.rand(3,3,3) &lt; 0.5] = np.NA<br>
&gt;&gt; &gt;&gt;&gt; a<br>
&gt;&gt; array([[[NA, NA, †0.11511708],<br>
&gt;&gt; † † † † [ 0.46661454, †0.47565512, NA],<br>
&gt;&gt; † † † † [NA, NA, NA]],<br>
&gt;&gt; † † † †[[NA, †0.57860351, NA],<br>
&gt;&gt; † † † † [NA, NA, †0.72012669],<br>
&gt;&gt; † † † † [ 0.36582123, NA, †0.76289794]],<br>
&gt;&gt; † † † †[[ 0.65322748, †0.92794386, NA],<br>
&gt;&gt; † † † † [ 0.53745165, †0.97520989, †0.17515083],<br>
&gt;&gt; † † † † [ 0.71219688, †0.5184328 , †0.75802805]]])<br>
&gt;&gt; &gt;&gt;&gt; np.mean(a, axis=-1)<br>
&gt;&gt; array([[NA, NA, NA],<br>
&gt;&gt; † † † †[NA, NA, NA],<br>
&gt;&gt; † † † †[NA, †0.56260412, †0.66288591]])<br>
&gt;&gt; &gt;&gt;&gt; np.std(a, axis=-1)<br>
&gt;&gt; array([[NA, NA, NA],<br>
&gt;&gt; † † † †[NA, NA, NA],<br>
&gt;&gt; † † † †[NA, †0.32710662, †0.10384331]])<br>
&gt;&gt; &gt;&gt;&gt; np.mean(a, axis=-1, skipna=True)<br>
&gt;&gt;<br>
&gt;&gt; /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2474:<br>
&gt;&gt; RuntimeWarning: invalid value encountered in true_divide<br>
&gt;&gt; † um.true_divide(ret, rcount, out=ret, casting=&#39;unsafe&#39;)<br>
&gt;&gt; array([[ 0.11511708, †0.47113483, † † † † nan],<br>
&gt;&gt; † † † †[ 0.57860351, †0.72012669, †0.56435958],<br>
&gt;&gt; † † † †[ 0.79058567, †0.56260412, †0.66288591]])<br>
&gt;&gt; &gt;&gt;&gt; np.std(a, axis=-1, skipna=True)<br>
&gt;&gt;<br>
&gt;&gt; /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2707:<br>
&gt;&gt; RuntimeWarning: invalid value encountered in true_divide<br>
&gt;&gt; † um.true_divide(arrmean, rcount, out=arrmean, casting=&#39;unsafe&#39;)<br>
&gt;&gt;<br>
&gt;&gt; /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2730:<br>
&gt;&gt; RuntimeWarning: invalid value encountered in true_divide<br>
&gt;&gt; † um.true_divide(ret, rcount, out=ret, casting=&#39;unsafe&#39;)<br>
&gt;&gt; array([[ 0. † † † †, †0.00452029, † † † † nan],<br>
&gt;&gt; † † † †[ 0. † † † †, †0. † † † †, †0.19853835],<br>
&gt;&gt; † † † †[ 0.13735819, †0.32710662, †0.10384331]])<br>
&gt;&gt; &gt;&gt;&gt; np.std(a, axis=(1,2), skipna=True)<br>
&gt;&gt; array([ 0.16786895, †0.15498008, †0.23811937])<br>
&gt;&gt;<br>
&gt;&gt; _______________________________________________<br>
&gt;&gt; NumPy-Discussion mailing list<br>
&gt;&gt; <a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
&gt;&gt; <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
&gt;&gt;<br>
&gt;&gt; Hi,<br>
&gt;&gt; That is great news!<br>
&gt;&gt; (Python2.x will be another email.)<br>
&gt;&gt;<br>
&gt;&gt; Python3.1 and Python3.2 failed with building &#39;multiarraymodule_onefile.o&#39;<br>
&gt;&gt; but I could not see any obvious reason.<br>
&gt;<br>
&gt; I&#39;ve pushed a change to fix the Python 3 build, it was a use<br>
&gt; of†Py_TPFLAGS_CHECKTYPES, which is no longer in Python3 but is always<br>
&gt; default now. Tested with 3.2.<br>
&gt; Thanks!<br>
&gt; Mark<br>
&gt;<br>
&gt;&gt;<br>
&gt;&gt; I had removed my build directory and then &#39;python3 setup.py build&#39; but I<br>
&gt;&gt; saw this message:<br>
&gt;&gt; Running from numpy source directory.<br>
&gt;&gt; numpy/core/setup_common.py:86: MismatchCAPIWarning: API mismatch detected,<br>
&gt;&gt; the C API version numbers have to be updated. Current C api version is 6,<br>
&gt;&gt; with checksum ef5688af03ffa23dd8e11734f5b69313, but recorded checksum for C<br>
&gt;&gt; API version 6 in codegen_dir/cversions.txt is<br>
&gt;&gt; e61d5dc51fa1c6459328266e215d6987. If functions were added in the C API, you<br>
&gt;&gt; have to update C_API_VERSION† in numpy/core/setup_common.py.<br>
&gt;&gt; † MismatchCAPIWarning)<br>
&gt;&gt;<br>
&gt;&gt; Upstream of the build log is below.<br>
&gt;&gt;<br>
&gt;&gt; Bruce<br>
&gt;&gt;<br>
&gt;&gt; In file included from<br>
&gt;&gt; numpy/core/src/multiarray/multiarraymodule_onefile.c:53:0:<br>
&gt;&gt; numpy/core/src/multiarray/na_singleton.c: At top level:<br>
&gt;&gt; numpy/core/src/multiarray/na_singleton.c:708:25: error:<br>
&gt;&gt; ĎPy_TPFLAGS_CHECKTYPESí undeclared here (not in a function)<br>
&gt;&gt; numpy/core/src/multiarray/common.c:48:1: warning: Ď_use_default_typeí<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/ctors.h:93:1: warning: Ď_arrays_overlapí<br>
&gt;&gt; declared Ďstaticí but never defined<br>
&gt;&gt; numpy/core/src/multiarray/scalartypes.c.src:2251:1: warning:<br>
&gt;&gt; Ďgentype_getsegcountí defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/scalartypes.c.src:2269:1: warning:<br>
&gt;&gt; Ďgentype_getcharbufí defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/mapping.c:110:1: warning: Ď_array_ass_itemí<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/number.c:266:1: warning: Ďarray_divideí defined<br>
&gt;&gt; but not used<br>
&gt;&gt; numpy/core/src/multiarray/number.c:464:1: warning: Ďarray_inplace_divideí<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/buffer.c:25:1: warning: Ďarray_getsegcountí<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/buffer.c:58:1: warning: Ďarray_getwritebufí<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/buffer.c:71:1: warning: Ďarray_getcharbufí<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/na_mask.c:681:1: warning:<br>
&gt;&gt; ĎPyArray_GetMaskInversionFunctioní defined but not used<br>
&gt;&gt; In file included from numpy/core/src/multiarray/scalartypes.c.src:25:0,<br>
&gt;&gt; †††††††††††††††† from<br>
&gt;&gt; numpy/core/src/multiarray/multiarraymodule_onefile.c:10:<br>
&gt;&gt; numpy/core/src/multiarray/_datetime.h:9:1: warning: function declaration<br>
&gt;&gt; isn‚t a prototype<br>
&gt;&gt; In file included from<br>
&gt;&gt; numpy/core/src/multiarray/multiarraymodule_onefile.c:13:0:<br>
&gt;&gt; numpy/core/src/multiarray/datetime.c:33:1: warning: function declaration<br>
&gt;&gt; isn‚t a prototype<br>
&gt;&gt; In file included from<br>
&gt;&gt; numpy/core/src/multiarray/multiarraymodule_onefile.c:17:0:<br>
&gt;&gt; numpy/core/src/multiarray/arraytypes.c.src: In function ‚VOID_getitem‚:<br>
&gt;&gt; numpy/core/src/multiarray/arraytypes.c.src:643:9: warning: passing<br>
&gt;&gt; argument 2 of ‚PyArray_SetBaseObject‚ from incompatible pointer type<br>
&gt;&gt;<br>
&gt;&gt; build/src.linux-x86_64-3.2/numpy/core/include/numpy/__multiarray_api.h:763:12:<br>
&gt;&gt; note: expected ‚struct PyObject *‚ but argument is of type ‚struct<br>
&gt;&gt; PyArrayObject *‚<br>
&gt;&gt; In file included from<br>
&gt;&gt; numpy/core/src/multiarray/multiarraymodule_onefile.c:44:0:<br>
&gt;&gt; numpy/core/src/multiarray/nditer_pywrap.c: In function<br>
&gt;&gt; ‚npyiter_subscript‚:<br>
&gt;&gt; numpy/core/src/multiarray/nditer_pywrap.c:2395:29: warning: passing<br>
&gt;&gt; argument 1 of ‚PySlice_GetIndices‚ from incompatible pointer type<br>
&gt;&gt; /usr/local/include/python3.2m/sliceobject.h:38:5: note: expected ‚struct<br>
&gt;&gt; PyObject *‚ but argument is of type ‚struct PySliceObject *‚<br>
&gt;&gt; numpy/core/src/multiarray/nditer_pywrap.c: In function<br>
&gt;&gt; ‚npyiter_ass_subscript‚:<br>
&gt;&gt; numpy/core/src/multiarray/nditer_pywrap.c:2440:29: warning: passing<br>
&gt;&gt; argument 1 of ‚PySlice_GetIndices‚ from incompatible pointer type<br>
&gt;&gt; /usr/local/include/python3.2m/sliceobject.h:38:5: note: expected ‚struct<br>
&gt;&gt; PyObject *‚ but argument is of type ‚struct PySliceObject *‚<br>
&gt;&gt; In file included from<br>
&gt;&gt; numpy/core/src/multiarray/multiarraymodule_onefile.c:53:0:<br>
&gt;&gt; numpy/core/src/multiarray/na_singleton.c: At top level:<br>
&gt;&gt; numpy/core/src/multiarray/na_singleton.c:708:25: error:<br>
&gt;&gt; ‚Py_TPFLAGS_CHECKTYPES‚ undeclared here (not in a function)<br>
&gt;&gt; numpy/core/src/multiarray/common.c:48:1: warning: ‚_use_default_type‚<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/ctors.h:93:1: warning: ‚_arrays_overlap‚<br>
&gt;&gt; declared ‚static‚ but never defined<br>
&gt;&gt; numpy/core/src/multiarray/scalartypes.c.src:2251:1: warning:<br>
&gt;&gt; ‚gentype_getsegcount‚ defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/scalartypes.c.src:2269:1: warning:<br>
&gt;&gt; ‚gentype_getcharbuf‚ defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/mapping.c:110:1: warning: ‚_array_ass_item‚<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/number.c:266:1: warning: ‚array_divide‚ defined<br>
&gt;&gt; but not used<br>
&gt;&gt; numpy/core/src/multiarray/number.c:464:1: warning: ‚array_inplace_divide‚<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/buffer.c:25:1: warning: ‚array_getsegcount‚<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/buffer.c:58:1: warning: ‚array_getwritebuf‚<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/buffer.c:71:1: warning: ‚array_getcharbuf‚<br>
&gt;&gt; defined but not used<br>
&gt;&gt; numpy/core/src/multiarray/na_mask.c:681:1: warning:<br>
&gt;&gt; ‚PyArray_GetMaskInversionFunction‚ defined but not used<br>
&gt;&gt; error: Command &quot;gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall<br>
&gt;&gt; -Wstrict-prototypes -fPIC -Inumpy/core/include<br>
&gt;&gt; -Ibuild/src.linux-x86_64-3.2/numpy/core/include/numpy<br>
&gt;&gt; -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core<br>
&gt;&gt; -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath<br>
&gt;&gt; -Inumpy/core/src/npysort -Inumpy/core/include<br>
&gt;&gt; -I/usr/local/include/python3.2m<br>
&gt;&gt; -Ibuild/src.linux-x86_64-3.2/numpy/core/src/multiarray<br>
&gt;&gt; -Ibuild/src.linux-x86_64-3.2/numpy/core/src/umath -c<br>
&gt;&gt; numpy/core/src/multiarray/multiarraymodule_onefile.c -o<br>
&gt;&gt; build/temp.linux-x86_64-3.2/numpy/core/src/multiarray/multiarraymodule_onefile.o&quot;<br>
&gt;&gt; failed with exit status 1<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
</div></div>Thanks for the prompt responses.<br>
<br>
That fixes the build problem for both Python3.1 and Python3.2.<br>
<br>
I got some test errors below but I guess you are working on those.<br>
<br>
<br>
Bruce<br>
<br>
<br>
<br>
$ python3 -c &quot;import numpy; numpy.test()&quot;<br>
<div class="im">Running unit tests for numpy<br>
</div>NumPy version 2.0.0.dev-965a5c6<br>
NumPy is installed in /usr/lib64/python3.2/site-packages/numpy<br>
Python version 3.2 (r32:88445, Feb 21 2011, 21:11:06) [GCC 4.6.0<br>
20110212 (Red Hat 4.6.0-0.7)]<br>
<div class="im">nose version 1.0.0<br>
</div>..............S.......EFF.....E............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...................................................................................................................................................................................................K..................................................................................................K......................K..........................................................................................................S......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./usr/lib64/python3.2/site-packages/numpy/lib/format.py:575:<br>

ResourceWarning: unclosed file &lt;_io.BufferedReader<br>
name=&#39;/tmp/tmpfmmo7x&#39;&gt;<br>
 †mode=mode, offset=offset)<br>
......................................................................................................................................................................................................................../usr/lib64/python3.2/subprocess.py:460:<br>

ResourceWarning: unclosed file &lt;_io.BufferedReader name=3&gt;<br>
 †return Popen(*popenargs, **kwargs).wait()<br>
/usr/lib64/python3.2/subprocess.py:460: ResourceWarning: unclosed file<br>
&lt;_io.BufferedReader name=8&gt;<br>
 †return Popen(*popenargs, **kwargs).wait()<br>
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................<br>

======================================================================<br>
ERROR: test_datetime_array_str (test_datetime.TestDateTime)<br>
----------------------------------------------------------------------<br>
Traceback (most recent call last):<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py&quot;,<br>
line 510, in test_datetime_array_str<br>
 † †assert_equal(str(a), &quot;[&#39;2011-03-16&#39; &#39;1920-01-01&#39; &#39;2013-05-19&#39;]&quot;)<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/numeric.py&quot;,<br>
line 1400, in array_str<br>
 † †return array2string(a, max_line_width, precision, suppress_small,<br>
&#39; &#39;, &quot;&quot;, str)<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py&quot;,<br>
line 459, in array2string<br>
 † †separator, prefix, formatter=formatter)<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py&quot;,<br>
line 331, in _array2string<br>
 † †_summaryEdgeItems, summary_insert)[:-1]<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py&quot;,<br>
line 502, in _formatArray<br>
 † †word = format_function(a[-i]) + separator<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/arrayprint.py&quot;,<br>
line 770, in __call__<br>
 † †casting=self.casting)<br>
TypeError: Cannot create a local timezone-based date string from a<br>
NumPy datetime without forcing &#39;unsafe&#39; casting<br>
<br>
======================================================================<br>
ERROR: test_datetime_divide (test_datetime.TestDateTime)<br>
----------------------------------------------------------------------<br>
Traceback (most recent call last):<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py&quot;,<br>
line 926, in test_datetime_divide<br>
 † †assert_equal(tda / tdb, 6.0 / 9.0)<br>
TypeError: internal error: could not find appropriate datetime inner<br>
loop in true_divide ufunc<br>
<br>
======================================================================<br>
FAIL: test_datetime_as_string (test_datetime.TestDateTime)<br>
----------------------------------------------------------------------<br>
Traceback (most recent call last):<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py&quot;,<br>
line 1166, in test_datetime_as_string<br>
 † †&#39;1959&#39;)<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/testing/utils.py&quot;,<br>
line 313, in assert_equal<br>
 † †raise AssertionError(msg)<br>
AssertionError:<br>
Items are not equal:<br>
†ACTUAL: b&#39;1959&#39;<br>
†DESIRED: &#39;1959&#39;<br>
<br>
======================================================================<br>
FAIL: test_datetime_as_string_timezone (test_datetime.TestDateTime)<br>
----------------------------------------------------------------------<br>
Traceback (most recent call last):<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/core/tests/test_datetime.py&quot;,<br>
line 1277, in test_datetime_as_string_timezone<br>
 † †&#39;2010-03-15T06:30Z&#39;)<br>
 †File &quot;/usr/lib64/python3.2/site-packages/numpy/testing/utils.py&quot;,<br>
line 313, in assert_equal<br>
 † †raise AssertionError(msg)<br>
AssertionError:<br>
Items are not equal:<br>
†ACTUAL: b&#39;2010-03-15T06:30Z&#39;<br>
†DESIRED: &#39;2010-03-15T06:30Z&#39;<br>
<br>
----------------------------------------------------------------------<br>
Ran 3063 tests in 37.701s<br>
<br>
FAILED (KNOWNFAIL=4, SKIP=2, errors=2, failures=2)<br>
<div><div></div><div class="h5"></div></div></blockquote><div><br>The 3.2 test errors aren&#39;t new. I&#39;d fix the tests except I&#39;m not sure if Mark wants to modify the datetime stuff instead.<br><br>Chuck<br></div>
<br></div>