Too often these days when people want to use a scientific study to bolster a political position, they utter the phrase, “It was peer reviewed” like a magical spell to shut off any criticism of a paper’s findings.
Worse, the concept of “peer review” is increasingly being treated in the popular discourse as synonymous with “the findings were reproduced and proven beyond a shadow of a doubt.”
This is never what peer review was intended to accomplish. Peer review functions largely to catch trivial mistakes and to filter out the loons. It does not confirm or refute a paper’s findings. Indeed, many scientific frauds have passed easily through peer review because the scammers knew what information the reviewers needed to see.
Peer review is the process by which scientists, knowledgeable in the field a paper is published in, look over the paper and some of the supporting data and information to make sure that no obvious errors have been made by the experimenters. The most common cause of peer review failure arises when the peer reviewer believes that the experimenters either did not properly configure their instrumentation, follow the proper procedures or sufficiently document that they did so.
Effective peer review requires that the reviewers have a deep familiarity with the instruments, protocols and procedures used by the experimenters. A chemist familiar with the use of a gas-chromatograph can tell from a description whether the instrument was properly calibrated, configured and used in a particular circumstance. On the other hand, a particle physicist who never uses gas-chromatographs could not verify it was used properly.
Shannon Love, “No One Peer-Reviews Scientific Software”, Chicago Boyz, 2009-11-28
December 10, 2009
QotD: Peer review
Comments Off on QotD: Peer review
No Comments
No comments yet.
RSS feed for comments on this post.
Sorry, the comment form is closed at this time.
