[SciPy-User] peer review of scientific software

Pauli Virtanen pav@iki...
Fri Jun 7 08:20:54 CDT 2013

Matt Newville <newville <at> cars.uchicago.edu> writes:
> But it appears to me
> that some people are under the impression that a) if code has unit
> tests it is bug free, and b) if code does not have unit tests, it is
> full of bugs.  Both are wrong.

Every scientist worth their salt of course tests whether their results
are robust. When this involves programming, the typical approach I've
seen is to run with inputs for which the expected outputs are known in
some other way, and compare results, check conservation laws etc.

The first issue of course is that this testing is done manually,
which is what you are inclined to do if you have not heard about
automated testing (which nobody bothers to teach you about during
your studies in non-IT fields).

For small code bases, this probably scales. For larger projects
or a longer time span, the amount of effort in testing increases.

The second issue is that typically what has been tested is not
recorded anywhere. (I have *never* seen anyone keep a programming
notebook, whereas a lab notebook is quite mandatory...) As a
consequence, you keep forgetting what you did. This wastes time,
and you cannot communicate to other people what you have actually
tested, even if they take you by your word.

Of course, you can maintain a set of test cases and expected results,
but this is essentially unit/functional testing.


I'd argue that a) and b) become more likely as the size of the code
base and development history grows bigger, unless you throw
proportional manpower to manual testing.

Manpower however does not remove the issue that being unable to
communicate to other people in a detailed way what you actually
did is unprofessional in science.

Pauli Virtanen

More information about the SciPy-User mailing list