[Numpy-discussion] Mysterious test_pareto failure on Travis

josef.pktd@gmai... josef.pktd@gmai...
Tue Sep 4 16:58:46 CDT 2012


On Tue, Sep 4, 2012 at 5:49 PM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
> On Tue, Sep 4, 2012 at 1:48 PM,  <josef.pktd@gmail.com> wrote:
>> On Tue, Sep 4, 2012 at 4:37 PM, Nathaniel Smith <njs@pobox.com> wrote:
>>> On Tue, Sep 4, 2012 at 9:17 PM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
>>>> On Tue, Sep 4, 2012 at 12:41 PM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
>>>>> On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
>>>>>> On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith <njs@pobox.com> wrote:
>>>>>>> The last two Travis builds of master have failed consistently with the
>>>>>>> same error:
>>>>>>>   http://travis-ci.org/#!/numpy/numpy/builds
>>>>>>> It looks like a real failure -- we're getting the same error on every
>>>>>>> build variant, some sort of problem in test_pareto. Example:
>>>>>>>   http://travis-ci.org/#!/numpy/numpy/jobs/2328823
>>>>>>>
>>>>>>> The obvious culprit would be the previous commit, which regenerated
>>>>>>> mtrand.c with Cython 0.17:
>>>>>>>   http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606
>>>>>>>
>>>>>>> What's weird, though, is that that commit passed just fine on Travis:
>>>>>>>   http://travis-ci.org/#!/numpy/numpy/builds/2313124
>>>>>>>
>>>>>>> It's just the two commits since then that failed. But these commits
>>>>>>> have been 1-line docstring changes, so I don't see how they could have
>>>>>>> possibly created the problem.
>>>>>>>
>>>>>>> Also, the test passes fine with python 2.7 on my laptop with current master.
>>>>>>>
>>>>>>> Can anyone reproduce this failure? Any ideas what might be going on?
>>>>>>
>>>>>> I made this:
>>>>>>
>>>>>> https://github.com/numpy/numpy/issues/424
>>>>>>
>>>>>> It was me who updated the Cython file. It seemed to be working. I've
>>>>>> added the issue
>>>>>> to the release TODO.
>>>>>
>>>>> Ok, here is how to reproduce the problem:
>>>>>
>>>>> 1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
>>>>> in the README:
>>>>>
>>>>> https://github.com/certik/numpy-vendor
>>>>>
>>>>> 2) run tests, you'll get:
>>>>>
>>>>> https://gist.github.com/3625509
>>>>
>>>> So the problem was actually introduced much earlier. Most probably it
>>>> has never worked
>>>> in 32bit Ubuntu 12.04. I tried for example this old commit:
>>>>
>>>> https://github.com/numpy/numpy/commit/3882d65c42acf6d5fff8cc9b3f410bb3e49c8af8
>>>>
>>>> and it still fails:
>>> [...]
>>>> But in any case, this seems to be a problem with the actual 32bit
>>>> Ubuntu 12.04 itself. So maybe something in gcc
>>>> has changed that now triggers the problem.
>>>
>>> To be clear, the mismatching value is:
>>>
>>>> /home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto()
>>> -> np.testing.assert_array_almost_equal(actual, desired, decimal=15)
>>> (Pdb) actual[1, 0]
>>> 52828779.702948704
>>> (Pdb) desired[1, 0]
>>> 52828779.702948518
>>>
>>> So they do match to 14 decimal points, and it's entirely possible that
>>> this is just a problem of the test being too stringent in requiring 15
>>> decimal points of match. Maybe the 32-bit GCC is spilling registers
>>> differently, tripping the famous x86 idiosyncrasy where register
>>> spilling triggers rounding. But I'd feel better if someone familiar
>>> with the pareto code could confirm.
>>
>> I don't understand this. Isn't assert_almost_equal testing decimals
>> not significant digits?
>
> Yes it is, see the docs:
>
>     The test is equivalent to ``abs(desired-actual) < 0.5 * 10**(-decimal)``.
>
> and the unit test is badly written, because that floating point
> number has maybe 14 or 15 good significant digits, but it simply does not have
> 15 digits after the decimal point, so in particular, that test is
> testing that the two
> floating point numbers are exactly equal.
>
>
> Here is a fix:
>
> https://github.com/numpy/numpy/pull/425
>
> Let me know if it is ok to merge it, so that we can work on other
> issues and have a working test suite.

I think  all the tests should be changed to use the new assert with
relative tolerance (similar to approx equal)
https://gist.github.com/3625943  shows also the other distributions
with 15 decimals when the values are larger than 10 for example

Josef


>
> Ondrej
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


More information about the NumPy-Discussion mailing list