Some Thoughts on Scientific Peer Review
It is no secret that the academic prestige of scientific articles increases exponentially when they are published in peer reviewed (refereed) journals. “The most sought-after professional publications,” notes Dr. Edward Barrett, past editor of the Journal of the Michigan Dental Association,“are those that contain refereed manuscripts. A refereed journal not only enhances the publication’s credibility but also that of the author(s). Some authors will submit articles only to refereed publications. In academia, many schools warrant tenure only on the basis of articles published in refereed journals. A publication’s credibility increases because a journal that is refereed gets a larger number of quality articles submitted for publications.1”
Peer reviewed status is now boldly proclaimed on the covers of more scientific and clinical journals than ever before. Even corporate magazines with obvious conflicts of interest are boasting peer review status.
What is a Peer-Reviewed or Refereed Journal?
Perhaps the simplest definition comes from Dr. Edward Barrett. He defines a refereed (peer reviewed) journal as “one that sends its scientific manuscripts to referees who advise the editor on the suitability of a publishing a given paper.1”
According to Dr. Michael Glick, past Editor of JADA, “The peer-review process is meant to help ensure that published manuscripts meet the scientific rigor of The Journal and that they are appropriate for the readership.2”
Current JADA editor J. Tim Wright emphasizes that “Presenting the truth, as best as it can be determined and based on the most reliable and reproducible available evidence, is the greatest responsibility of journals publishing biomedical sciences.3” He notes that achieving this goal has been a challenge for centuries—since the inception of scientific publishing.
The peer review process, says Rosy Hoskins of MIT4, “is a relatively recent innovation in the history of scientific publication. The first journal (which is still in print!) was launched in 1665 by the Royal Society in London, (Phil Trans R Soc B), while peer review as we know it began in the mid-1970s.” She noted that Watson and Crick’s famous paper in Nature magazine on the Double Helix structure of DNA could not have been peer reviewed, as it was published in 1953. Nevertheless John Maddox, a subsequent editor of the magazine affirmed that “its correctness is self-evident.” The reason for peer review, however, is that correctness is NOT self-evident.
Types of Peer Review
According to Anne Helmenstine, founder of Science Notes, there are three main types of peer review5:
-
-
-
-
-
- Single-blind review, where reviewers know the identities of the authors, but the authors do not know the identities of the reviewers.
- Double-blind review, where both the authors and reviewers remain mutually anonymous.
- Open peer review, where the identities of both authors and reviewers are disclosed in order to promote transparency and collaboration.
-
-
-
-
Ms. Helmenstine notes that anonymous reviews reduce bias but allow collaboration, whereas open reviews are more transparent, but increase bias.
Most scientific publications follow a double-blind protocol. In February of 2021, JADA announced the implementation of this protocol.6
Creating a Peer Review Process for Scientific Journals
Creating a peer review system is much more complex than it seems. There are as many ways of conducting peer review as there are publications. “No authoritative list of refereed journals exists in any discipline”, says former Texas Dental Journal Doug Willingham, “because the process varies.2”
I was tasked with creating a peer review process for the New York State Dental Journal as chairman of NYSDA’s Council on Publications in the late nineties. Developing this process was an investment in countless hours of study and Council deliberation. Ultimately, we created guiding principles for authors, editors and reviewers, a method for gauging the system’s effectiveness, and a balanced panel of reviewing experts.
While the peer review process seems idyllic and beneficial, there are some serious concerns for any peer review system. The peer review processing of scientific articles takes far longer, and difficulties may arise with reviewers that could impact publishing operations. There is no question that a peer review process means more work for the editorial staff. Staff has to ensure that the publication does not incur a reputation for “slowness” in order to compete with rival publications for scientific articles. JADA has a panel of 1800 reviewers.2 I cannot even fathom how gargantuan the task must be to juggle numerous authors and their submissions with the selection of appropriate reviewers from this panel!
The greatest concern of any peer review system is the promotion of “elitism,” orthodoxy, or inherent bias. It is critical that publications ensure that reviewers do not represent any particular region, institution or school of thought. The review process is unscientific by nature and may inadvertently encroach on the free dissemination of scientific information.
In addition to having the opportunity to thoroughly study and implement a peer review system with my colleagues, I have been an active participant of the peer review system—both as an author and as a reviewer. Authors should never take offense to criticism—it is never meant to be “personal,” but as advice that could improve research, articles, and writing.
Serving as Peer Reviewer
It has been my privilege to serve as a reviewer for the Journal of Oral Implantology—the official publication of the American Academy of Implant Dentistry. I have reviewed more than 85 papers as a panel reviewer. The JOI uses a double-blind approach toward reviewing papers, so authors and reviewers do not know each other. The review process involves numerous evaluation criteria, and the reviewer is allowed to make comments directly to the author or only to the editor. I view my role as assisting the editor, so I make all my comments only to the editor.
When I first started reviewing papers I was immediately impressed at how many references were included with each paper. I know how difficult it is to write academic papers and lots of references meant hours of study (or so I thought). As I continued to review papers, I became less impressed as I began to scrutinize the actual content. I quickly became a “hard-nose,” meaning that I freely spewed out my unvarnished opinions. I thought initially that my bluntness would result in termination as a panel reviewer–but here I am 85 papers later and they are still coming! My insights, I believe, are unique because I have such an unusual restorative background. I am impressed that the editor, Dr. Jim Ruthowski, appreciates what I have to say. I hope it helps him make the best decisions for publishing articles, whether or not he agrees with my advice.
As a reviewer, I have some interesting general observations about the papers that I have reviewed:
- Best Papers: The best papers are generally retrospective analyses of previously published research.
- Agendas: There is always a reason why research was conducted and why authors want to publish articles. Professionals like to believe that the motivation is the pure pursuit of science, but this is rarely the case. The quest for publishing is often motivated by confirmation of bias, profit-dri
ven outcomes, or furthering an academic career. Tina Hesman Saey of Science News magazine, notes that there are several reasons why research goes wrong. These reasons include the aforementioned agendas as well as fuzzy math, sloppy protocols, sins of omission, mistakes, lack of consideration for biologic variability, erroneous conclusions and fraud.7 When viewed through the proper lens, reviewers can often identify the agendas. - Blind Acceptance of Dogma: Tom Siegfried of Science News holds academia accountable: “”The Scientific Method of Testing Hypotheses stands on a flimsy foundation,” he exclaims, “standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making decisions.” It is my observation that academics commonly transform hypotheses into gospel fact without a shred of corroborating evidence. As a result, they promote the blind acceptance of misconceptions. Misconceptions in the area of full coverage restorative dentistry, for example, are responsible for many difficulties that clinicians are having with crown and bridgework. My textbook, which is in the process of being published, addresses these misconceptions in great detail.
- Poorly Designed Studies: Original research is usually incomplete because the sample sizes are too small, or the study did not have enough duration to make meaningful conclusions. I believe that at least five years is the minimum for any clinical research study. Many papers are submitted from Europe where doctoral candidates have to be published in order to receive their doctorates (a bad idea!). These doctoral candidates are not around long enough to conduct five years of study.
- Poor case selection: Poor case selection is a common problem with case reports. I recently reviewed a paper that featured a technique of customizing a special stent to add bone grafts above the inferior alveolar nerve for the placement of implants. The treatment worked. However, not enough bone was actually placed that would allow the placement of implants with enough depth to support the restoration. It was not clear that the restoration could succeed, even though the bone grafts were initially successful. Case reports, like this one, usually do not include years of follow-up to validate case success.
- Follow-up: In fact, most clinician lecturers and researchers are not in the habit of presenting ANY follow-up of clinical cases! Without follow-up, how can readers be sure treatments were successful? Follow-up must include X-Rays post-treatment and at least five years post-treatment for verification that the treatment worked. The esthetic result on the day of insertion means nothing. If a case is esthetic but ultimately fails, should that treatment be emulated by readers? I think NOT.
- Excessive Demonstrations of Academic Prowess: Some authors use complex vocabulary that the average reader cannot understand. Authors commonly take for granted that readers are familiar with sophisticated research equipment and protocols. The reality is that most readers are clinicians who have no idea what these authors are talking about. Articles should be easy for readers to understand, but not over-simplified. Authors have to explain how and why sophisticated equipment and protocols are used. As Albert Einstein once said: “If you can’t explain it simply, you don’t understand it well enough.”
While the peer review process is far from perfect, its primary value is to promote and validate scientific learning. Dentistry is a learned profession, and continuous learning is the hallmark of professionalism. Everyone involved in the peer review process learns something. Researchers learn how to improve their articles, reviewers learn by reviewing papers, and the editor learns something from both the authors and the reviewers. Let’s not forget the readers, as they are ultimate beneficiaries of the peer review process. The featured articles measure up to high standards and are designed to deliver the highest quality education.
1Barrett, Edward; “What is a Refereed Journal?”; Journal of the Michigan Dental Association; September, 1988.
2Glick, Michael, DDS; “Peer review: An inexact but essential part of scientific publishing;” JADA; Volume 138, Issue 5p568-570-571May 2007; DOI: 10.14219/jada.archive.2007.0212.
3Wright, J. Tim DDS, MS; “Peer Review: A culture of science and service;” JADA; Volume 155, Issue 9; p725-72; 6 September 2024; DOI: 10.1016/j.adaj.2024.07.003.
4Hosking, Rosy; with contributions from the MIT CommLab Diana Chien, Mike Orella, Akshata Sonni, Chris Gerry and Geoffrey North; “Peer Review: Past, Present and Future: an imperfect but still relevant pillar of scientific publishing;” https://mitcommlab.mit.edu/broad/commkit/peer-review-a-historical-perspective/
5Helmenstine, Ann; “Understanding Peer Review in Science;” May 9, 2023; https://sciencenotes.org/understanding-peer-review-in-science/.
6Announcements 23 February 2021; JADA; “The Journal of the American Dental Association implements a double-blind peer review process;” https://www.sciencedirect.com/journal/the-journal-of-the-american-dental-association/about/announcements/the-journal-of-the-american-dental-association.
7Saey, Tina Hesman; “12 Reasons Research Goes Wrong;” Science News; January 24, 2015; https://www.sciencenews.org/article/12-reasons-research-goes-wrong.
8Siegfried, Tom; “Odds are its wrong;” Science News; March 27, 2010; Vol. 177 #7; p. 26; https://www.isepp.org/pdf%20and%20doc%20files/Odds%20Are,%20It’s%20Wrong%20-%20Science%20News.pdf
You may also like
