Friday, June 7, 2013


The Peer Review Problem and a Proposed Solution

Peer review

The peer review process is not as flawed as many think.  It is needed to keep professional journals and granting agencies functioning at a high level.  When working as intended, it is a benefit to scientists, saving time from having to sort through useless material that should never have been published, and making research money go to those most likely to do productive research.

The feared failures such as reviews that are uninformed, biased, eliminating work from competitors or other lapses are minimal according to some research on the topic done years ago.  Peer reviewers may be overworked and usually unpaid volunteers.  They usually are active publishers in the field they review.  But complaints arise.

I have thought about the problem.  The following overlong suggestion for a solution was formulated about seven years ago.  It was after about ten years of retirement, keeping up my interest in biological sciences, and it occurred to me that I could have been part of the solution.  It is too late for me now, but it might provoke someone to do something about it.

Joe Engemann      June 7, 2013

December 14, 2005

I just finished lunch as I looked a bit at the AARP Bulletin.  A brief train of thought brought me to a solution to the editorial and peer review problem facing top journals.  The solution seemed to be an equivalent organization to the Good Housekeeping Seal of Approval or Consumers Union/Consumers Reports.  Perhaps NSF would fund startup or AARP could help advance such a project.

The problem

Editors of scientific journals do not have the time and/or the competence to properly consider and evaluate all submissions.  Reviewers face similar problems.  Peer reviewers are seldom true peers of workers in all aspects of a discipline.  Errors that creep in to a discipline are unlikely to be seen by either those that let the error in or by those they taught. 

The goal

Establish a core of retired volunteer reviewers in various specialties to establish an independent journal review program.  Panels of three or more should produce evaluations of Science, Nature, and the top (one to three) specialty journals in each major subdiscipline.  The evaluations would reflect standards of the area, with some input from outsiders to the discipline.  A sampling process could be used unless sufficient reviewers or panels are available to evaluate all papers and editorial content. 

The review program could be a clearing house for individuals to submit complaints and/or for journals to refer complaints regarding their perception of failings of particular published materials.  Emphasis on the complaint generated review process could reduce unnecessary reviews of journals that have an effective high quality program or output.  A method of evaluating complaints would be desirable, particularly if excess numbers were generated.  Perhaps a quantitative score could be developed to prioritize review choices to those getting the highest relative score (relative to readership and available reviewers or better criteria).

Choice of reviewers

Retirees that normally read the journal
Over 65 or retired over 5 years
Over ten years since editorial board or other service for a journal or related entity
Degree and/or employment, publication, or other credentials in the discipline

Tenure of reviewers

One year renewable for five years upon unanimous agreement of other panel members
A second five year term could ensue if no replacements are available
Removable by petition of one or more panel members if validated by the majority of each of two of the three most closely related panels

Oversight board

One member elected from each panel to serve on a roster from which a six member board to be selected by lot – three from the panels making up the third having most review activity, two from the next third, and one from the remaining third having least activity
Remaining elected members to serve as alternates (by request of board chair or member needing replacement) and/or replacements (next on alphabetical list of those elected)
Function of board to be administration or administrative oversight and development of rules or by-laws as necessary


Volunteers could cover their own expenses as a public service
Reviewers should not be paid, except for expenses if available, to maintain independence and avoid conflicts of interest
Foundations might provide support either long term or to get started
The Western Michigan University Evaluation Center might provide advice or funding
Possible funding from the National Science Foundation for secretarial and office expenses in start-up phase
Possible funding from membership fees from journals if mechanism avoiding conflict of interest is possible
Possible funding from newsletter reporting evaluations available by subscription; perhaps libraries might find use in evaluating acquisitions

Independent results might be of use to granting agencies (NSF, NIH, Foundations, etc) to evaluate grants made for factors producing quality publications (grant support of publications could be in database of reviews).  The public might find it a valuable way to evaluate giving to non-profit organizations sponsoring research.

Submission for AARP (bulletin or journal)

Is peer review failing?

Retired members of scientific societies, do you still read your journals?  If you do, your input is desired.  Do you think you could provide independent review of the validity of research published in your specialty?

A disgruntled emeritus biology professor maintains two leading general scientific journals have been publishing research articles based on research violating both valid principles of research and basic research published in specialty journals.  The professor wants AARP to survey members to see if journals they read also show such a failing of peer review.

The professor thinks the independent input of retired scientists could help correct some of the deficiencies of the peer review process based on the following reasoning.  The information explosion along with specialization required to produce research in molecular biology has resulted in most publications being produced by those lacking the broad background prevalent in most researchers of two generations ago.  The peer review process is great to preserve quality, but it also preserves errors that become ingrained in those winning a competitive race to publish.  They are slow, unwilling, or incapable of seeing views in conflict with their own work.  They have learned some of the errors as accepted principles of their discipline.

The best example is the error of calculating relationships of animals in an evolutionary tree of life based on faulty molecular clock studies.  Life cycle duration was a possible variable noted in a major publication starting molecular clock studies.  It was ignored because it was unknown how to deal with it, it had little effect on closely related groups, and the oversight became institutionalized in the field.  The error was further compounded by using single chemical compounds (variability in speed of change in different groups was widely documented), too small a sample size (less than the minimum number shown to be needed), and data selection.

It seemed a horrendous error to replace the results of studies developed through a century of work using comparative anatomical, embryological, and biochemical, as well as paleontological and other studies, on the basis of an oversimplified molecular clock study.

Is the professor a kook?  Or do you agree something needs to be done and/or would be willing to participate in reviewing one or more journals in your field?  Send your response to the AARP, Peer review survey, - - - - - .

6/7/2013   The above was not sent.  But, is anybody out there ready to do something about it?

No comments:

Post a Comment