Not long ago, an article was "published" making the remarkable claim of genetic evidence showing the existence of a population of "Bigfoot" being (a) real and (b) some sort of hybrid between humans and some unknown non-human primate. "Published" is in quotation marks because of the odd manner in which the paper was finally presented: Melba Ketchum, the author, bought a little-known web journal that was apparently going out of business anyhow, renamed it De Novo, and published her own paper. She claims that her paper had already passed peer review in the earlier incarnation of the journal (the Journal of Advanced Zoological Exploration in Zoology), but that the journal's lawyers would not agree to it being published; she also claims that the decision to publish her article was not her own, but that of the (unnamed) editorial staff of De Novo. She further claims that she had to take these steps in order to make this important work available to the public, although it could have been made available at lower cost both to herself and the public if she had simply uploaded the pdf file to a web page or preprint server. Unsurprisingly, this article, which would have been controversial enough without the theatrics, has gained little support outside the community of those who were at least already convinced that Bigfoot is a real, biological entity, not just a cultural phenomenon.
I am not going to address the details of her genetic analysis because I lack the background to do so. (Incidentally, her own qualifications for interpreting the genetic tests are also shaky. Dr. Melba Ketchum is a veterinarian, which, although it does relate to biology, involves very different training than that of a research biologist. The actual genetic tests were apparently carried out by commercial labs at her expense, with her contribution being the interpretation of the results.) Instead, I will concentrate on some misconceptions regarding peer review that I noticed on a prominent cryptozoology blog. My comments here will largely recapitulate comments I made there under a pseudonym.
First of all, let me clear up what peer review is not. Peer review is not the acceptance of an idea by one or a few people with Ph.D.s. Passing peer review is also not a golden ticket that makes an idea scientifically respectable. Papers published in obscure journals tend to remain obscure; even papers in well-known journals may be overlooked, or their true significance may not be understood right away. Papers are often published that are controversial or speculative, and on occasion they are published in spite of the expressed reservations of the editors. An excellent example is the publication of an article on the (since discredited, but never really respectable) "Torah code" in the journal Statistical Science.
Peer review is all about trust and accountability. There are two levels at which trust is important.
(1) Most fundamentally, of course, scientists have to trust that any mistakes are honest mistakes, not deliberate fraud. Even deliberate fraud will eventually be uncovered by subsequent experiments, but the amount of time, expense, and even danger may be greatly increased by fraudulent data.
Peer review is really not particularly good at exposing deliberate fraud unless the fraud is especially amateurish; even in those cases it may go overlooked for a surprisingly long time.
What, then, guards against scientific fraud? Two things. The first is that real data will almost certainly be obtained later that will sharply contradict any fraudulent data. The second is that the resulting loss of reputation can be toxic to a career, making it impossible to get papers published, obtain grants, or hold any academic position. In a recent prominent case, a German physicist lost his Ph.D. because he was judged unworthy of it -- for a fraud that took place after he earned his doctorate. Just earning the Ph.D., to say nothing of the professional work that follows, is hard enough that no sane person would carelessly endanger it.
How does this apply to the Ketchum case? Well, she is not a research scientist; her job as a veterinarian is not really at risk, nor is any scientific reputation. With nothing at stake, she will naturally be regarded with some additional suspicion. She might still have been given consideration, but she could not afford doing anything that compounded the suspicion.
(2) Three hundred years ago, it was still possible to remain well-informed on the current state of all branches of science. The success of science in the intervening centuries has come at a cost, though; today it is barely possible to remain up-to-date on a tiny subfield of a particular discipline. In consequence, it has become necessary to have a "spam filter". That is what peer review really does; a paper that passes peer review makes it to the inbox, until then the paper is in the spam folder.
Of course, spam filters are not all alike. Some are too strict and put good messages in the spam folder; some are too permissive and let spam through; some do both; and some get it about right. Likewise with the peer review conducted by the various journals. Each journal has a slightly different scope -- what the journal is about -- and one thing peer review has to do is determine if the paper was submitted to the right journal. Different journals also have different length requirements (some specialized for quick notes, and others for lengthy reviews of the current state of knowledge about particular topics). Finally, they differ on the number of reviewers, the pool of reviewers they seek out, and the instructions given to the reviewers. This is largely what makes one journal more prestigious than another. This is why one journal's peer review is not transferable to another.
Yet this is precisely what Ketchum claims for her paper. She claims (though does not present evidence) that her paper "really" passed peer review before, so there was no need to repeat the process. The only sense in which this is true is that the journal she bought seems to have barely been hanging on to existence anyhow, with no apparent reputation and (from what I could find on the Wayback Machine) no clear statement of scope and no instructions for reviewers.
The upshot is that Ketchum either completely fails to understand what peer review is about or (worse) does understand but is deliberately attempting to deceive those who do not. In either case, she will find it hard to earn respect from people who have dedicated their lives to science.
No comments:
Post a Comment