United Kingdom Parliament’s Committee announced new inquiry into peer review

Science and Technology Committee of the House of Commons (United Kingdom’s Parliament) announced the launch of an inquiry into peer review, and invited for evidence on “the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication”.

The Committee called for submissions on all aspect of the process: measures to strengthen peer review, the value and use of peer review, differences in peer review regarding various scientific disciplines and countries across the world, selection of reviewers in particular as the volume of multi-disciplinary research increases, the impact of Information and Communication Technologies, and possible alternatives to peer review. The deadline to make these submission was Thursday 10 March 2011.

More details at http://www.parliament.uk/business/committees/committees-a-z/commons-select/science-and-technology-committee/news/110127-new-inquiry—peer-review/

Is there any known public report regarding this UK Parliaments inquiry into peer review?

Advertisements

Costs, Prices, and Revenues in Journals Publishing

Publishing costs, prices, and revenues might be found in several studies. Revenues per article vary from $1,000 to $20,000 (Ginsparg, 2001; 2003). Prices per article, charged to authors, as Article Processing Charge (APC) in Open Access journals, vary from $500 to $ 3000 (Waltham, 2006; p. 128). According to Ginsparg (2001), electronic publishing costs (with no print product) were in the range of $500-$1000 per article, in 2001. Watham (2005) affirms that the average cost of learned societies on-line publishing (i.e. after removing the print costs), in 2004, was £956.25 (about $1,708) per article and £97.5 (about $193) per page. More details can be found at http://www.iiisci.org/journal/sci/Costs.pdf

There is a huge difference between revenues, prices, and costs per article. The lowest reported cost is US$ 500 per article, and the largest reported revenue per article is US$ 20,000. Is this huge difference related to some of the problems reported regarding Peer Reviewing and to some decisions regarding which journals to index and which not to index? Are there Peer Reviewing methodologies more costly than others as to justify revenue of about 40,000% more than the minimum reported cost per article?

 References

Ginsparg, P., 2001,   ‘Creating a global knowledge network,” in Electronic Publishing in Science, UNESCO HQ, Paris, 19-23 Feb 2001, http://people.ccmr.cornell.edu/~ginsparg/blurb/pg01unesco.html (Accessed on May 19, 2012).

Ginsparg, P., 2003, “Alternatives to peer review II: can peer review be better focused?” in F. Godlee and T. Jefferson (Eds), Peer Review in Health Sciences, London: BMJ Publishing Group; pp312-321.

Waltham, M., 2005, JISC: Learned Society Open Access Business Models, 184, Springdale Road, Princeton , NJ 08540, USA. Available at http://www.marywaltham.com/JISCReport.pdf (Accessed on August 4th, 2008).

Waltham, M., 2006, “Learned society business models and open access,” Open Access: Key Strategic, Technical and Economic Aspects, (Neil Jacobs, ed.); Oxford, England: Chandos Publishing; pp. 121-129.

Meanings of “Peer” and “Peer Review”

Is there any accepted meaning of “Peer” and “Peer Review”? Should not this question be consensually answered before defining and implementing any peer reviewing process?

Weller (2002) asserts that “Since editorial peer review is a process, its definition can and does vary according to how the process in envisioned.” There is a general obvious definition: “evaluation by one’s peers”, but the meaning of “one’s peers” differs according different editorial policies and according different universities’ regulations. Furthermore, since the meaning “evaluation” depends, by definition, on the epistemological values of the reviewer and the editor (or conference organizers), how can we know if these values do not contradict each other, and are in agreement with those of the author or, at least, with those epistemological values shared by the disciplinarians of the respective disciplines? What about inter-disciplinary articles? Who are adequate peers for its respective reviewing? What epistemological values are to support the evaluation process?

DeBakey (1990) affirms that “ ‘Peer reviewing’ has become a stock term, but –he asks– is a reviewer of a manuscript…always a peer: a person who has equal standing with another, as in rank, class or age?” (p. 347, cited by Weller, 2002, p. 16) So, if we define “peer” as a person who has equal standing with another, as in rank (equal standing of academic rank, for example) few conference organizers or editors can rightfully and with certainty say that their journals or proceedings are “peer” reviewed.

Stumph (1980) asserts that “For most advanced scientists only a few or no peers exist. In their research new areas are explored, often with special techniques and approaches. There is thus a high probability that one or several aspects of a proposal will not be appreciated by the judging ‘quasi-peers’…for advanced scientists, a competent review cannot be achieved unless a reviewer who is working in the same field with similar amount of experience is consulted.” Consequently, “the closest scientific peer is a competitor. Even Though reviewers try to be fair, nobody likes his or her programs for original ideas to be screened and judged by a real or potential competitor.” (p. 822).

Combining his arguments with those of Stumph (1980), Chubin (1990) concludes that “Scientists are at the mercy of peer review systems that may offer neither ‘peers’ nor ‘review’.” (p. 194; emphasis added)

More on this issue at http://www.iiis.org/nagib-callaos/meaning-of-peer-review

References

Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy; New York, State University of New York Press.

DeBakey, L., 1990, Journal peer reviewing. Anonymity or disclosure? Archives of Ophthalmology, 108(3), pp. 345-349. Cited in Weller, A. C., 2001, Editorial Peer Review, its Strength and Weaknesses, Medford, New Jersey, p.16.

Stumph, W. E., 1980, “ ‘Peer’ review,” Science 207, 22 February, pp. 822-823.

Weller, A. C., 2001, Editorial Peer Review, its Strength and Weaknesses, Medford, New Jersey.

Comments, made to the initial post on “Peer Review” are congruents and complement each other.

The comments received in this blog (up to May 16th, 2012) include ideas related to possible (at least partial) solutions to the problem of the frequently reported low effectiveness level of peer review. Among these ideas that potentially might increase the effectiveness of peer review are the following (we hope we are not changing in an essential way the meaning of the text used by the different commentators because of taking them out of their context):

Nadav Har’El proposes that Everybody can post *any* paper, without any process whatsoever. Readers who come across this paper can “review” this paper… Google Scholar’s ability to find highly-cited papers is a good start, because citations act as a good sign of someone thought this paper was important.”

Lbrieda proposes a collaborative effort and to explore “blogging and commenting” as a way of improving peer review effectiveness, and explain why in his opinion this means would not lower the quality level of what is being published in the blog. In our opinion Lbrieda might be right, but in the worst case that he is not, “blogging and commenting” is certainly an effective first phase in the publishing process. In our opinion, what Lbrieda is proposing is congruent with what Nadav Har’El suggests as a potential (and partial, in our opinion) solution. What Lbrieda is proposing might be conceived as a way of implementing what Nadav Har’El suggests.

Fabrizio agrees with Lbrieda proposal regarding a “collaborative effort” and informs about an actual implementation of this idea but for those articles that has already been published. In our opinion, something similar to what Fabrizio is informing about and he is proposing might be also be implemented for the initial informal pre-publication, or in any intermediate phase in the publication process. Fabrizio notes that what has been implemented “doesn’t supersede peer review process but it is sort of complementary” with the potential of improving its effectiveness or ameliorating some of its flaws. In our opinion, what Fabrizio is proposing is congruent with what Lbrieda and Nadav Har’El are proposing. The difference is that Fabrizio proposes it as post-print review and Lbrieda and Nadav Har’El proposes a similar solution but as some kind of pre-print review.

Ottoblau estimate that it is not feasible to make radical changes in Peer Review and proposes “a step-by-step change of peer reviewing.” As a first step he proposes to eliminate the “cover letter” because he considers that it “is the first step towards non-scientific evaluation of science.” Then he affirms “Let’s apply pressure to the editors that they do their job and read at least the abstract. An avalanche of emails to chief editors could easily fix the problem.” In our opinion, even if Ottoblau reasoning makes sense, it will contribute very little to solve the real problem of peer review ineffectiveness. Some would even think that it would amplify the problem because the editors will have less support in choosing adequate reviewers. The editors are not supposed to be experts in all disciplines, sub-disciplines, and sub-sub-disciplines included in the journal of which they are editors.

Jeremyhornephd reminds us that knowledge quality is the foundation of peer review. Consequently, we need to formulate what knowledge quality really means, and this is more of a process than a result.” It might be a good idea to start another post about the meaning “Knowledge Quality,” and what kind of scholars should engage in the process of generating some light with regards to this issue. In our opinion, this is an intellectual endeavor in which philosophers, scientists, engineers, epistemologists, etc. should get involved with.

Ted affirms that journal publishing is “outmoded, hyper-expensive medium.” On the other hand “The current problem with the blog environment is that everyone and anyone can blog or publish online.” Then he also proposes collaborative effort processes. He affirms that “If everyone is writing, then we need to share the load of reviewing with everyone, or else the system can’t keep up… Papers and authors should have status…. we need to be considering ranking mechanisms that improve the visibility of the best work instead of trying to prevent the publication of the worst.” Great idea in our opinion, and we have the means to implement it. This idea does not exclude any of the other comments that have been made. On the contrary: it complements and it is in syntony with them.

Anatoly Sorokin affirms that “the problem is not how we are reviewing papers, the problem is why we publish so much.” Then he affirms as a potential solution “We need to separate process of writing from process of fundability evaluation.” This, in our opinion, requires changes in academic’s and grant organisms’ cultures, policies and promotional procedures.  Is this feasible? If not, how long and what kind of resources would it take to make it potentially feasible? We think that while trying to change academic’s and grants organisms’ cultures and policies, we can improve the effectiveness of peer review by following suggestions made by other commentators. Ted, for example, while agreeing in 100% with Anatoly Sorokins, affirms that “Nothing should ever be prevented from being published. But only the GOOD papers should be rewarded through academic advancement or whatever.”

EVESQUE Pierre affirms that “One of the main problems with peer reviewing is that there is not so much material to study… The referee reports and the discussions with the editors are considered as usual correspondence, which cannot be published… This is the main problem against scientific understanding and scientific evaluation of peer reviewing.” He described and effort he made during 10 years trying to explore, though what we might call Action Research (researching action of researching and publishing), and after investing (wasting?) time and effort dealing with established cultures he estimate that he had a partial success. He describes, via direct experience, the lack of motivation of academics and the scientific communities in reviewing processes, especially in post-publishing reviewing. EVESQUE Pierre is actually trying to find solutions, implement them, and research the respective processes in order to improve what has been implemented. This is, in our opinion, the kind of Action-Research, combined with Action-Learning and (probably implicit) Action-Design, which might very probably be effective in enquiries oriented to increase the effectiveness of peer review.

We have been following a similar approach along the last 10 years.  Our approach and the process we have been following are described in http://www.iiis.org/nagib-callaos/peer-review/. The potential solutions provided up to the present in the commentaries made in this blog are congruent among themselves and with what we have suggested in the mentioned document. We think that it might be very helpful if actions are taken in the context of Action-Research in the domain of peer review in order to improve its effectiveness.

Nagib Callaos

 

Peer Review: Is it effective? Is it possible to improve its effectiveness? Are there other means to evaluate research?

 (Draft in progress)

In a survey of members of the Scientific Research Society, “only 8% agreed that ‘peer review work well as it is’.”[1] “If peer review was a drug it would never be allowed onto the market,” affirmed Drummond Rennie[2], deputy editor of the Journal Of the American Medical Association and who intellectually provided support for the international congresses of peer review that have been held, since 1989, every four years. If peer review was a drug, it “would not get onto the market because we have no convincing evidence of its benefits but a lot of evidence of its flaws.”[3] Richard Smith (2006, p. 116) also affirmed that regarding peer review there is “more evidence of harm than benefit…[and] Studies so far have shown that it is slow, expensive, ineffective, something of a lottery, prone to bias and abuse, and hopeless at spotting errors and fraud.”[4] Few days ago, Carl Zimmer reported in the New York Time that, according to a study made by PubMed data base, the number of articles retracted from scientific journals increased from (three) in 2000 to 180 in 2009[5]. 6000% of increment in 10 years! One author, Naoki Mori, had published about 30 papers that later has been retracted. Just one journal, the International Journal of Cancer had to retract seven articles authored, or co-authored, by Naoki Mori. “Nobody had noticed the whole thing was rotten,” said Dr. Fang (professor at the University Of Washington School of Medicine), referring to the huge number of retractions by prestigious journals, and related to the same author by prestigious journals. This “Sharp Rise in Retractions Prompts Calls for Reform”[6]

Studies and many experienced editors of prestigious journals have been reporting this problem during at least, three decades. We reported about a significant number of these studies and editors’ experience-based opinions in a more detailed article (http://www.iiis.org/nagib-callaos/peer-review). In this article we 1) described the main weaknesses of peer reviewing processes, 2) identified the objectives of peer review, 3) proposed potential solutions and 4) resumed the process by means of which we identified what might mitigate the reported weaknesses and what might improve the effectiveness of peer review. Among the potential solutions we proposed is to substitute the traditional lineal model of classical printed publications in hard copies with a non-lineal model, for electronic publications, based on cybernetics loops of continuous negative and positive feedback loops. This lineal model does not necessarily exclude some linearity for hard copy publications. In the traditional model the activities of research, writing the results, reviewing and potentially publishing them are done basically in series. But, with the current web information systems, and especially with web 2.0 concepts and technologies, all the above mentioned activities might be done basically in parallel. Accordingly we proposed to develop a web-based information system that would support the proposed cybernetic model. A detailed description of this system is included in above mentioned article[7] (http://www.iiis.org/nagib-callaos/peer-review). We developed about 70% of the proposed software but the Global Recession inhibited the flow of the financial funds that were required to finish the software development and to implement it.

But, the main ideas of the mentioned cybernetic model might certainly be implemented by the current public social media. In our opinion, blogging is one of the adequate means to 1) collect and share reflections and experiences regarding the weaknesses of peer review and its potential solutions, and 2) to provide critical judgments regarding the proposed approach. Consequently, we are initiating this blog as a first step that might produce valuable comments, new information, and the creation of related blogs including, hopefully, research blogs on this very important issue that is requiring urgent solutions or, at least, research oriented to identify potential solutions.

An increased number of concerned scholars are insisting in the importance and urgency of making research regarding “peer review,” “research evaluation,” or “quality assurance of research publications.” Richard Smith, for example, referring to “peer review,” affirms that “Despite being central to the scientific process it was itself largely unstudied until various pioneers including Stephen Lock, former editor of the BMJ [British Medical Journal], and Drummond Rennie, deputy editor of JAMA [Journal of the American Medical Association] — urged that it could and should be studied. Studies so far have shown that it is slow, expensive, ineffective, something of a lottery, prone to bias and abuse, and hopeless at spotting errors and fraud. The benefits of peer review have been much harder to establish.” It is time to make another kind of studies, explorations, and experimentation: those oriented to improve peer review or to substitute it with other means of “research evaluation” or “research publishing quality assurance.” Accordingly, we presented an exploratory proposal[8] and we have the willingness to put our two cents in giving a first step in this (potentially collective) blog and in the process of fostering the creation of more research blogs regarding the improvement of “peer review” or its substitution by other methods of “research evaluation.”

This blogging activity is being started by one or two persons with the hope that more authors will be added as contributors or editors, and more related blogs will be created regarding this issue.

This blog, along with the potential set of related blogs, will very probably be hybrid one: 1) sharing information and reflections, 2) referring to related documents, and 2) doing research via blogging as one of the research means.


[1] Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy; New York, State University of New York Press, p. 192.

[2] Cited by Smith, R, 2010, “Classical peer review: an empty gun,” Breast Cancer Research 2010, 12(Suppl 4):S13, (accessed at http://breast-cancer-research.com/content/12/S4/S13), p1.

[3] Ibid.

[4] Smith, R, 2006, “The trouble with medical journals,” Journal of the Royal Society of Medicine, Vol. 99, March, 2006, p. 116 (accessed at http://jrsm.rsmjournals.com/content/99/3/115.full.pdf)

[5] Zimmer, C., 2012, “A Sharp Rise in Retractions Prompts Calls for Reform,” The New York Times, April 16, 2012 (accessed on April 19, 2012 at http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?_r=1&comments

[6] Ibid.

[7] Callaos, N., 2011, Peer Reviewing: Weaknesses and Proposed Solutions, accessed on April 22, 2012 at http://www.iiis.org/nagib-callaos/peer-review/

[8] Ibid.