Some Schmo wrote: ↑Tue Nov 15, 2022 1:26 am
The vast majority of scientific hypotheses are thrown out. Not the same for peer reviewed theories.
Don't get me started on peer review. Ah, too late.
Peer review is not some kind of gold standard that ensures things are reliable. Peer review is simply review conducted by peers, as in peer-to-peer, as opposed to having review by some central authority, like a world panel of experts. So who selects these "peers"? That's the thing.
If you ever publish a scientific paper, you are marked for life as a "peer" on that topic. Journal editors will pursue you almost as relentlessly as the Church of Scientology, to get you to review other papers submitted to their journals.
You will eventually be getting e-mails every week or two from some journal or other, many of them obscure journals that you have never even noticed, let alone actually read (though some will be prestigious journals whose attention will at first seem flattering until you realize that they are also just trying to get free work from you). They will want you to invest a couple of hours, for free and anonymously, to give a dispassionate recommendation on whether or not they should publish this manuscript. If the manuscript is not fantastic as it is, but could potentially be improved into something worth publishing, they expect you to give them free advice, which they can pass on to the manuscript authors, on exactly how to improve the paper.
This will all happen even if you are not especially brilliant and your published paper was nothing special. The fate of many papers will nonetheless be decided by you and one or two other randos like you, who happen to have published something not too far removed from the topic, at some point.
You might be tickled by some unimportant feature of a paper and enthusiastically recommend it, while carelessly overlooking its disastrous flaws. Or you might misunderstand something through ignorance and send in a damning assessment of a brilliant breakthrough. Unless another reviewer disagrees with you violently, or something in your recommendation sounds sketchy enough to the editors that they decide to consult someone else just on spec, your totally fallible judgement will get a bad paper published, or a good one rejected. That's peer review.
Sometimes you don't even have to have published a paper to get dinged as a peer reviewer. Maybe your doctoral advisor offloaded some of their reviewing work onto you, telling some particular journal that you were a sufficient expert. Thereafter that journal will be happy to consult you again directly, repeatedly. Since your advisor's vouching for your expertise wasn't public information like a publication, I'm not sure whether any other journals will get wind of you and start hounding you, too; but it wouldn't surprise me so much if journals shopped lists of names to each other. The whole peer review process bears an uncomfortably distinct resemblance to e-mail spam.
It's still the best system we have. If a good paper gets rejected, oh well. The authors will send it to another journal and it will get published eventually. It might end up published in a less prominent journal, but if it's really such a good paper then people will eventually notice it, wherever it is. If a bad paper gets published, oh well. People will read it and realize how bad it is. Few peer reviewers are geniuses, and many are too busy to spend as much time as authors and editors might wish, but most are quite competent and take the job of reviewing papers seriously as a professional duty. Apart from giving thumbs up or down, reviewers often improve papers substantially with their revision suggestions.
Peer review is a good standard, but it's a minimal standard, not a gold standard. Tons of stuff that gets published in high-caliber peer-reviewed journals is wrong. Not infrequently it's wrong in its conclusions, but very often it's wrong in thinking that they are new and important. Back when libraries had shelves, it was a physics joke to observe that the number of journal articles published had been growing exponentially over time for so long that soon the volumes would be filling shelves at a speed faster than light, and this would be possible because no information was being transmitted.
Major scientific theories that have been tested and applied for decades by thousands of researchers around the world are really reliable. Even if they turn out to be wrong in important ways, they won't just have been completely wrong. But that kind of status is far beyond peer review. If that kind of status for a scientific theory is like being a Hall of Fame athlete, having passed peer review is like having been on a team in high school. It's something, but not all that much.
I was a teenager before it was cool.