[SciPy-User] peer review of scientific software

Matthew Brett matthew.brett@gmail....
Sun Jun 2 00:47:17 CDT 2013


Hi,

On Sat, Jun 1, 2013 at 8:29 PM,  <josef.pktd@gmail.com> wrote:

> following the link from the PLOS editorial statement
> http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124
>
> I think the entire premise "are research findings false" is completely
> misguided. It just continuous the magic 0.05 tradition.

I don't think it is as simple as that.   For example, one of the
studies I cited before was only able to replicate 6 / 53 'landmark'
studies in hematological oncology.

http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

"Clearly there are fundamental problems in both academia and industry
in the way such research is conducted and reported. Addressing these
systemic issues will require tremendous commitment and a desire to
change the prevalent culture. Perhaps the most crucial element for
change is to acknowledge that the bar for reproducibility in
performing and presenting preclinical studies must be raised."

I've been canvassing my colleagues over the last year or so about what
replication rate they would guess in brain imaging, and the answers
are rather variable, but have a mean around 30 percent.  These
estimates are from people running brain imaging centers or very
experienced in the field.

If these estimates are correct, the waste is enormous, overwhelming.

> Otherwise it's like calculus. Some need it most of their life, the
> other ones forget about it as soon as the exams are over.
> (But you cannot learn calculus and statistics by doing, and there is
> only limited amount of time students have. More statistics please.)

The person who is trying to do work in Excel, that should be done in a
programming language, needed that training.  They will be doing slower
work. and make more errors for the lack of a small amount of training.
 For sure the tech-smart guy or gal in the lab makes a big difference,
but not every lab has such a person, and it's common (believe me) for
researchers who don't know this stuff to assume it's only for nerds
and that it only slows down getting real work done.  That's largely a
function of lack of training in how easy it is to make mistakes, and
therefore the necessity of using tools to reduce mistakes and improve
transparency.

I've also noticed that when people are not comfortable with their
tools, they often fail to notice obvious statistical issues that they
would normally expect to spot at once.  Here's an obvious example from
brain imaging:

http://www.edvul.com/voodoocorr.php

So, if you teach people statistics and you don't teach them how and
when to program, and they have to do anything other than point and
click in SPSS, you'll often get bad statistics none the less.

Cheers,

Matthew


More information about the SciPy-User mailing list