Editor of scholarly journal talks peer review process

In light of the recently published hoax studies, a Portland State professor and academic journal editor said the success of the hoax articles indicates a failure of the peer review system as a whole, not a failure of those who authored them.

PSU Professor of History and Editor for the scholarly journal, Pacific Historical Review, Marc Rodriguez, explained “there was a failure—not so much on the part of the faculty member that did this act of satire or rebellion—but it was a failing of the peer review system to do its job.”  

Following the publication of the hoax studies, an anonymous op-ed was published in Vanguard, critiquing the actions of the professors who submitted the articles.

“This created quite a bit of discussion among journal editors about the process and what peer review means,” Rodriguez said. “What I thought was interesting was none of the responses tended to come down in favor of the views expressed by the anonymous faculty.”

Another point made in the op-ed is that academics are under stress to publish even when they’re busy.

“It’s my view, and I think the view of most editors, that if faculty are stressed it doesn’t mean they should do an inadequate job of review,” Rodriguez said. “It means that they should say no to the journal or not review something.”

Most scholarly journals are edited by faculty recruited for the editorial positions by academic associations. When a professor or graduate student does research and preparation for an article, they send it to a journal. The article is then stripped of any self reference so it can go through a blind review.

“We read it and take it seriously,” Rodriguez said. “And if we don’t think that it’s up to our standards or if we don’t think it’s actually serious…it’s rejected.”

Rejection rates vary by journal. Prestigious journals such as Nature come in just under an 8 percent acceptance rate, while other journals which have fewer submissions or different selection criteria have higher rates.

If the journal editors or the editorial review board think the draft is serious and meets their standards, they then seek out scholars in the article’s particular field to undertake the peer review process.

“[Editors] ask professors around the country to take a look at the draft which has no one’s name on it,” Rodriguez said, explaining that blind review is intended to remove bias from the review process and to mitigate the influence from a professor’s longstanding network.

This network is “about who’s powerful in the academy,” Rodriguez said. “For example, Ivy League faculty tend to have power…people listen to them and CNN wants to interview them.”

A study done by a team at Google showed the double-blind peer review process does in fact have an impact on how positively peer reviewers receive academic papers. This supports the idea that academics might receive more favorable reviews based on their name power alone, according to The American Council on Science and Health.

The reviewers prepare a so-called readers report for the journal. “Some of these are wonderfully detailed and helpful; some are less so,” Rodriguez said. If a reviewer submits an inadequate review or a one-sentence response, editors are not likely to push the article forward. If all of the reviews are weak, the journal will search for more reviewers.

After an article gets reviewed, the author gets commentary back from the editor on what to fix, such as factual errors or weak sources, giving the author the chance to revise and resubmit their work.

“Sometimes this process can go through four [to five] cycles of revision,” Rodriguez said, adding that he has seen authors who have done three to seven separate review processes to get an article published.

When the peer reviewers have decided the article fits specific disciplinary requirements, they inform the editors they believe it’s ready for publishing.

“If the referees are in support, the editor [sometimes in consultation with the Editorial Board depending on policy] makes a decision to accept the article, or after review, reject it,” Rodriguez explained. “Sometimes professors don’t like the review process because it’s taking too long and they’ll go to a journal that is known to have looser standards.”

These “looser” journals are known among academics to fast-track the process, and have published with less scrutiny and fewer reviewers than what is standard.  

“[The process] might take a while, but in the end the more [the article’s] been tested and [the more] hard questions have been asked in the review process, the better the final [product],” Rodriguez said. “More undergraduates will read it in their classes and more people will cite it for the next 25 or 30 years because it was well done.”