Peer review
The peer review process is not as flawed as many think. It is needed to keep professional journals and granting agencies functioning at a high level. When working as intended, it is a benefit to scientists, saving time from having to sort through useless material that should never have been published, and making research money go to those most likely to do productive research.
The feared failures such as reviews that are uninformed, biased, eliminating work from competitors or other lapses are minimal according to some research on the topic done years ago. Peer reviewers may be overworked and usually unpaid volunteers. They usually are active publishers in the field they review. But complaints arise.
I have thought about the problem. The following overlong suggestion for a solution was formulated about seven years ago. It was after about ten years of retirement, keeping up my interest in biological sciences, and it occurred to me that I could have been part of the solution. It is too late for me now, but it might provoke someone to do something about it.
Joe Engemann June 7, 2013
I just finished lunch as I looked a bit at the AARP
Bulletin. A brief train of thought
brought me to a solution to the editorial and peer review problem facing top
journals. The solution seemed to be an
equivalent organization to the Good Housekeeping Seal of Approval or Consumers
Union/Consumers Reports. Perhaps NSF
would fund startup or AARP could help advance such a project.
The problem
Editors of scientific journals do not have the time and/or
the competence to properly consider and evaluate all submissions. Reviewers face similar problems. Peer reviewers are seldom true peers of
workers in all aspects of a discipline.
Errors that creep in to a discipline are unlikely to be seen by either
those that let the error in or by those they taught.
The goal
Establish a core of retired volunteer reviewers in various
specialties to establish an independent journal review program. Panels of three or more should produce
evaluations of Science, Nature, and the top (one to three)
specialty journals in each major subdiscipline.
The evaluations would reflect standards of the area, with some input
from outsiders to the discipline. A
sampling process could be used unless sufficient reviewers or panels are
available to evaluate all papers and editorial content.
The review program could be a clearing house for individuals
to submit complaints and/or for journals to refer complaints regarding their
perception of failings of particular published materials. Emphasis on the complaint generated review
process could reduce unnecessary reviews of journals that have an effective
high quality program or output. A method
of evaluating complaints would be desirable, particularly if excess numbers
were generated. Perhaps a quantitative
score could be developed to prioritize review choices to those getting the
highest relative score (relative to readership and available reviewers or
better criteria).
Choice of reviewers
Retirees that normally read the journal
Over 65 or retired over 5 years
Over ten years since editorial board or other service for a
journal or related entity
Degree and/or employment, publication, or other credentials
in the discipline
Tenure of reviewers
One year renewable for five years upon unanimous agreement
of other panel members
A second five year term could ensue if no replacements are
available
Removable by petition of one or more panel members if
validated by the majority of each of two of the three most closely related
panels
Oversight board
One member elected from each panel to serve on a roster from
which a six member board to be selected by lot – three from the panels making
up the third having most review activity, two from the next third, and one from
the remaining third having least activity
Remaining elected members to serve as alternates (by request
of board chair or member needing replacement) and/or replacements (next on
alphabetical list of those elected)
Function of board to be administration or administrative
oversight and development of rules or by-laws as necessary
Funding
Volunteers could cover their own expenses as a public
service
Reviewers should not be paid, except for expenses if
available, to maintain independence and avoid conflicts of interest
Foundations might provide support either long term or to get
started
The Western
Michigan University
Evaluation Center
Possible funding from the National Science Foundation for
secretarial and office expenses in start-up phase
Possible funding from membership fees from journals if mechanism
avoiding conflict of interest is possible
Possible funding from newsletter reporting evaluations
available by subscription; perhaps libraries might find use in evaluating
acquisitions
Independent results might be of use to granting agencies
(NSF, NIH, Foundations, etc) to evaluate grants made for factors producing
quality publications (grant support of publications could be in database of
reviews). The public might find it a
valuable way to evaluate giving to non-profit organizations sponsoring research.
Submission for AARP (bulletin or journal)
Is peer review failing?
Retired members of scientific societies, do you still read
your journals? If you do, your input is
desired. Do you think you could provide
independent review of the validity of research published in your specialty?
A disgruntled emeritus biology professor maintains two
leading general scientific journals have been publishing research articles
based on research violating both valid principles of research and basic
research published in specialty journals.
The professor wants AARP to survey members to see if journals they read
also show such a failing of peer review.
The professor thinks the independent input of retired
scientists could help correct some of the deficiencies of the peer review
process based on the following reasoning. The information explosion along with
specialization required to produce research in molecular biology has resulted
in most publications being produced by those lacking the broad background
prevalent in most researchers of two generations ago. The peer review process is great to preserve
quality, but it also preserves errors that become ingrained in those winning a
competitive race to publish. They are
slow, unwilling, or incapable of seeing views in conflict with their own
work. They have learned some of the
errors as accepted principles of their discipline.
The best example is the error of calculating relationships
of animals in an evolutionary tree of life based on faulty molecular clock
studies. Life cycle duration was a
possible variable noted in a major publication starting molecular clock
studies. It was ignored because it was
unknown how to deal with it, it had little effect on closely related groups,
and the oversight became institutionalized in the field. The error was further compounded by using
single chemical compounds (variability in speed of change in different groups
was widely documented), too small a sample size (less than the minimum number
shown to be needed), and data selection.
It seemed a horrendous error to replace the results of
studies developed through a century of work using comparative anatomical,
embryological, and biochemical, as well as paleontological and other studies,
on the basis of an oversimplified molecular clock study.
Is the professor a kook?
Or do you agree something needs to be done and/or would be willing to
participate in reviewing one or more journals in your field? Send your response to the AARP, Peer review
survey, - - - - - .
6/7/2013 The above was not sent. But, is anybody out there ready to do something about it?
No comments:
Post a Comment