Peer Review: Is it effective? Is it possible to improve its effectiveness? Are there other means to evaluate research?

 (Draft in progress)

In a survey of members of the Scientific Research Society, “only 8% agreed that ‘peer review work well as it is’.”[1] “If peer review was a drug it would never be allowed onto the market,” affirmed Drummond Rennie[2], deputy editor of the Journal Of the American Medical Association and who intellectually provided support for the international congresses of peer review that have been held, since 1989, every four years. If peer review was a drug, it “would not get onto the market because we have no convincing evidence of its benefits but a lot of evidence of its flaws.”[3] Richard Smith (2006, p. 116) also affirmed that regarding peer review there is “more evidence of harm than benefit…[and] Studies so far have shown that it is slow, expensive, ineffective, something of a lottery, prone to bias and abuse, and hopeless at spotting errors and fraud.”[4] Few days ago, Carl Zimmer reported in the New York Time that, according to a study made by PubMed data base, the number of articles retracted from scientific journals increased from (three) in 2000 to 180 in 2009[5]. 6000% of increment in 10 years! One author, Naoki Mori, had published about 30 papers that later has been retracted. Just one journal, the International Journal of Cancer had to retract seven articles authored, or co-authored, by Naoki Mori. “Nobody had noticed the whole thing was rotten,” said Dr. Fang (professor at the University Of Washington School of Medicine), referring to the huge number of retractions by prestigious journals, and related to the same author by prestigious journals. This “Sharp Rise in Retractions Prompts Calls for Reform”[6]

Studies and many experienced editors of prestigious journals have been reporting this problem during at least, three decades. We reported about a significant number of these studies and editors’ experience-based opinions in a more detailed article (http://www.iiis.org/nagib-callaos/peer-review). In this article we 1) described the main weaknesses of peer reviewing processes, 2) identified the objectives of peer review, 3) proposed potential solutions and 4) resumed the process by means of which we identified what might mitigate the reported weaknesses and what might improve the effectiveness of peer review. Among the potential solutions we proposed is to substitute the traditional lineal model of classical printed publications in hard copies with a non-lineal model, for electronic publications, based on cybernetics loops of continuous negative and positive feedback loops. This lineal model does not necessarily exclude some linearity for hard copy publications. In the traditional model the activities of research, writing the results, reviewing and potentially publishing them are done basically in series. But, with the current web information systems, and especially with web 2.0 concepts and technologies, all the above mentioned activities might be done basically in parallel. Accordingly we proposed to develop a web-based information system that would support the proposed cybernetic model. A detailed description of this system is included in above mentioned article[7] (http://www.iiis.org/nagib-callaos/peer-review). We developed about 70% of the proposed software but the Global Recession inhibited the flow of the financial funds that were required to finish the software development and to implement it.

But, the main ideas of the mentioned cybernetic model might certainly be implemented by the current public social media. In our opinion, blogging is one of the adequate means to 1) collect and share reflections and experiences regarding the weaknesses of peer review and its potential solutions, and 2) to provide critical judgments regarding the proposed approach. Consequently, we are initiating this blog as a first step that might produce valuable comments, new information, and the creation of related blogs including, hopefully, research blogs on this very important issue that is requiring urgent solutions or, at least, research oriented to identify potential solutions.

An increased number of concerned scholars are insisting in the importance and urgency of making research regarding “peer review,” “research evaluation,” or “quality assurance of research publications.” Richard Smith, for example, referring to “peer review,” affirms that “Despite being central to the scientific process it was itself largely unstudied until various pioneers including Stephen Lock, former editor of the BMJ [British Medical Journal], and Drummond Rennie, deputy editor of JAMA [Journal of the American Medical Association] — urged that it could and should be studied. Studies so far have shown that it is slow, expensive, ineffective, something of a lottery, prone to bias and abuse, and hopeless at spotting errors and fraud. The benefits of peer review have been much harder to establish.” It is time to make another kind of studies, explorations, and experimentation: those oriented to improve peer review or to substitute it with other means of “research evaluation” or “research publishing quality assurance.” Accordingly, we presented an exploratory proposal[8] and we have the willingness to put our two cents in giving a first step in this (potentially collective) blog and in the process of fostering the creation of more research blogs regarding the improvement of “peer review” or its substitution by other methods of “research evaluation.”

This blogging activity is being started by one or two persons with the hope that more authors will be added as contributors or editors, and more related blogs will be created regarding this issue.

This blog, along with the potential set of related blogs, will very probably be hybrid one: 1) sharing information and reflections, 2) referring to related documents, and 2) doing research via blogging as one of the research means.


[1] Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy; New York, State University of New York Press, p. 192.

[2] Cited by Smith, R, 2010, “Classical peer review: an empty gun,” Breast Cancer Research 2010, 12(Suppl 4):S13, (accessed at http://breast-cancer-research.com/content/12/S4/S13), p1.

[3] Ibid.

[4] Smith, R, 2006, “The trouble with medical journals,” Journal of the Royal Society of Medicine, Vol. 99, March, 2006, p. 116 (accessed at http://jrsm.rsmjournals.com/content/99/3/115.full.pdf)

[5] Zimmer, C., 2012, “A Sharp Rise in Retractions Prompts Calls for Reform,” The New York Times, April 16, 2012 (accessed on April 19, 2012 at http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?_r=1&comments

[6] Ibid.

[7] Callaos, N., 2011, Peer Reviewing: Weaknesses and Proposed Solutions, accessed on April 22, 2012 at http://www.iiis.org/nagib-callaos/peer-review/

[8] Ibid.

 

17 responses to “Peer Review: Is it effective? Is it possible to improve its effectiveness? Are there other means to evaluate research?

  1. Nadav Har'El May 3, 2012 at 12:24 pm

    It’s easy to find flaws in the peer review process. It’s slow, for highly-competitive journals or conferences, the cutoff between acceptance and rejection is often almost aritrary, and it doesn’t catch many cases of bad science. But the question is, what is the alternative?

    As a person who was a reviewer of many papers, I admit that there are indeed things that peer review doesn’t filter out, but know what kind of crap (sorry for the language) it does filter out. It filters out bad and even patently ignorant ideas, badly written papers, old ideas that everyone (but the author) already knows about. Basically, it leaves only *interesting* papers. If I read a peer-reviewed journal or attend a peer-reviewed conference, I know I’m not wasting my time because each one of the papers was judged by my peers to be *interesting*, so I would with high degree of probability find it interesting too. I’d hate to lose this advantage of peer review.

    Another advantage of peer review is that authors make a bigger effort because of it (and I know this from my experience as an author). On your website/blog/whatever, you can send any sort of half-baked idea. If this was the accepted practice, most researchers would stop here – and not go the extra mile to better research their idea and better document it. But with peer review, authors are encouraged to improve the quality of their writing and the quality of their arguments (and tests, or whatever, depending on field), so as to convince their peers. Moreover, they know they need to work harder for more competitive journals/conferences, to be even more convincing. And authors want to reach the more competitive journals because they have more readers, and because people still use this as promotion criteria.

    How can we keep these advantages without the current peer review process?

    One possibility is to have a (sorry for the blashpemy) “like”-oriented, or in other words large-scale peer review process: Everybody can post *any* paper, without any process whatsoever. Readers who come across this paper can “review” this paper – grading it on quality, interestingness, novelty, truth, writing quality, and so on, and also leaving a textual review if their wish. One who wishes to find interesting papers can then find them by limiting himself by these grades.

    Google Scholar’s ability to find highly-cited papers is a good start, because citations act as a good sign of someone thought this paper was important. I often use “Google Scholar” to find interesting papers, without caring which journal the result came from (or what kind of peer review process this journal used). But citation is a very crude instrument – and very slow to adapt (typically taking several years to get the first estimate on the interestingness of a paper). It would have been nice to have a much faster estimate of reader-judged quality of papers.

  2. lbrieda May 4, 2012 at 11:12 pm

    Nadav, while I agree with you on the basic merits of official peer review, I also believe this system is broken as is. In fact, it is what motivated me to start my own scientific-themed blog, http://www.particleincell.com/blog. My main issues with the current peer review model is that it doesn’t promote a true discussion of the subject matter. The system is very archaic.The rest of the world has moved to a new collaborative standard, yet scientific publication is still set in the stone age. Consider this post as an example. Let’s say your comment was published in a journal. If I wanted to respond, I would first have to write a formal letter following a prescribed format, send it to the editor for a review, wait few weeks (months?) to get it published, etc… It’s a huge hassle! And since it’s so hard to comment, people hardly ever do, and we end up taking stuff published in journals to be the truth. I had a professor in college who spent a lot of time analyzing papers. It was quite amazing how many mistakes he found even in papers published in respectable journals like Science.

    But in the online world it is easy to comment. If I write something that is incorrect, people will respond. And the comments stay with the blog post, allowing a future reader to see all counterpoints, etc.. And when it comes to the quality of content, I totally agree, there is a lot of crap on the internet. But there is also a lot of good stuff – take Wikipedia for an example. I don’t agree that posting online somehow makes one put in less effort than publishing in a more traditional format. Just as an example, one of my other blogs has over 1500 email subscribers. The last thing I want to do is flood their email boxes with some garbage! If I did that I would surely get back more hell than from the one, two, or three reviewers that will ever see my manuscript.

    Anyway, this is a really good discussion. It think the tide is starting to turn, with even some big name journals now exploring blogging and commenting. We’ll see what the future holds.

    • Egidijus May 6, 2012 at 5:44 pm

      The topic is worth to be discussed, however simple solutions did not exist. Let’s imagine, Sir Newton has published an article about universal gravitational constant in the online world. 99 % of comments at that time would be negative- everybody knows that a stone falls faster than feather. Internet analysts are hidden under name and many of them doesn’t feel responsibility for their comments. Peer reviewer is a person with address, position, etc.

      • lbrieda May 19, 2012 at 5:00 am

        I don’t think that’s necessary true. One of the main criticisms of commenting is that most comments are junk but that’s not actually my experience from the few websites I run (of course there is spam but that gets filtered out automatically by akismet). Also, critical comments are actually quite helpful because they give you a chance to gain a better confidence in your theory. It’s not an easy task to explain your point to somebody who may not agree with your points, but once you can do that you will gain better confidence. At least that’s my case. Not saying I like people disagreeing with me, but sometimes being forced to defend my view allows me to put aside the little doubts that are in my mind (or make me realize I was wrong, that’s also often the case).

    • Fabrizio May 8, 2012 at 1:24 pm

      Dear Ibireda and all,

      I agree on your point about the collaborative efforts. Another way to overcome this difficulty is to setup a commentary blog on any of the refereed paper published in a sector. After many discussion with collegues, we have built a prototype webapplication (http://www.youastro.org) which does that in a very straightforward way (for now, only for astrophysical papers published on main international journals). Note, however, that this doesn’t supersede peer review process but it is sort of complementary, an open space where scientist can discuss papers as they do in meetings, but in a written, open way. Strong or weak points, cases of plagiarism, any other relevan comments may be posted, even anonymously, lightly moderated by a Board of Editor which ensure that comments remain clean of any unappropriate content.

      We beleive an application like that may contribute,on the long run, to better papers, without renouncing completely to the peer review process.

      Regards
      F.

  3. ottoblau May 6, 2012 at 2:46 pm

    We will not change the peer review process radically. Radical changes (revolutions) happen when the majority of a society has no hope. The hopeless guys leave the society when possible. Leaving the science society is possible. Therefore, I do not expect a radical change of the peer review process.

    I suggest a step-by-step change of peer reviewing. First things first. Let’s start with the cover letter:

    Almost all journals require an abstract in plain language, brief and comprehensive. Why do editors request a cover letter? Is their education below average so they cannot understand an abstract? Which additional information could go into a cover letter? I can only think about non-scientific information like:

    “Hey buddy, we know each other since years. Treat my manuscript well.”

    “I already published sooo many articles in this field. How dare you to even think about not sending this one out for review?”

    “This paper opens the door to a complete revolution in treatment of cancer.” (If this would be true and the manuscript contained scientific evidence, why would it not be written in the plain, brief and comprehensive abstract?)

    I am sure you find some more non-scientific things to be written in a cover letter. But I do not believe you could figure a scientific argument that should be written in a cover letter only.

    Another argument for cover letters is that editors are so busy that they need a brief summary to evaluate manuscripts. A summary (abstract) of the abstract (summary)? 200 words are too many for an editor to read?

    Many of you probably think that the cover letter is not the biggest problem of peer reviewing. The cover letter is the first step towards non-scientific evaluation of science. Let’s apply pressure to the editors that they do their job and read at least the abstract. An avalanche of emails to chief editors could easily fix the problem.

    Let’s fix this first problem and do the next step afterwards.

  4. Ted May 6, 2012 at 10:32 pm

    My problem is less to do with peer review, and more to do with the nature of written journals themselves. Journal publication was ideal as a minimally peer reviewed process when any sort of publication was painful and difficult. Journals were literally the fastest way to communicate your ideas with the scientific community, and the existing peer review process was designed to work within that context and only needed to separate out less rigorous academic work. What got through was usually at least genuine, if not altogether insightful or novel.

    Since generally the only people with access to journals are scientists working at a large university with an ample library budget, the only people even reading the information had to be in a university. Therefore a minimal filter was necessary. Today, with online media like wikipedia available to everyone, more and more the idea of knowledge being restricted to those who will pay through the nose for it is no longer tenable.

    The current problem with the blog environment is that everyone and anyone can blog or publish online. So that means that the online world is much much noisier, and sorting the wheat from the chaff is much more demanding. In the new world, when we think of a review process we need to not even be thinking about journal publication. It’s an outmoded, hyper-expensive medium. We can no longer think in terms of a review process that prevents publication.

    If everyone is writing, then we need to share the load of reviewing with everyone, or else the system can’t keep up. The number of times a seminal paper is cited currently gives an idea of how important that paper is. What we need is an online system where links between papers and the sources of those papers are tracked automatically. Every source would be instantly readable, and free. Papers and authors should have status. Having papers referenced gives status to the author and the paper. Having a higher education or specialization in the field (e.g. Ph.D.) adds to the rank of author, but not the paper, and only in the field of expertise. Rather than a “like” counter, each author is allowed to “favorite” their favorite papers. Further, everyone can see that favorites list. The higher the author’s status, the more papers the author can keep as favorites. That way, it is not a simple anonymous like or dislike.

    We could do all this because we need to be considering ranking mechanisms that improve the visibility of the best work instead of trying to prevent the publication of the worst work.

    Ted

  5. Anatoly Sorokin May 12, 2012 at 5:07 am

    I think the problem is not how we are reviewing papers, the problem is why we publish so much. Since establishment of grant science the focus of publishing moves from “look what a great idea I have” to “hey I need this out to get more money”. Definetly in the further case the slow and careful reviewing process benefits the author, as it allows to finalize and polish the idea. Ideal for latter case would be the publishing in the blog and report its measure in Mb to the funding body.

    The peer review was designed to support science, as an attempt to understand the nature, and the major reason it is failing now is that we are on a way from science to “information production industry”. How many times you’ve thought I would rather combine those two papers in to one? How many times you’ve done this?

    To keep peer reviewing as a pillar of science we need to understand why there are so many papers published and can we convince people that one well thought and comprehensive paper as a result of five years PhD is better than eleven one-experiment publications. We need to separate process of writing from process of fudability evaluation. Otherwise we will end up in the situation as described in Leo Szilard’s 1948 story “The Mark Gable Foundation” as a recipe to retard the scientific process.

    Anatoly

    • Ted May 13, 2012 at 7:01 pm

      I agree with Anatoly 100%. Publish or perish as the axiom goes. We have ended up in this mode where people feel the need to mass produce individual papers to both get funding or to advance to associate or full professorship. That means science is now a competitive environment rather than a collaborative one. It also means that since publication can take years, that frequently there is overlap and poor coordination between people doing research on the same exact topic, and any communication takes place only outside that publication window. Lord only knows the destructive effect of requiring your competitors sign off on your work first, which is what is happening in our current reviewing process. I feel that if our reward system paid for quality rather than quantity then things might be better. Nothing should ever be prevented from being published. But only the GOOD papers should be rewarded through academic advancement or whatever.

      • Fabrizio May 14, 2012 at 1:25 pm

        Maybe the increase of the number of published papers is unavoidable, because we are studying so many sectors of science and the number of scientists is increasing (on a time scale of a few decades, of course). But indeed, the “publish or perish” axiom mentioned by Ted and others is doing its load of damage, this is evident. I don’t see a clear exit strategy for this, if not rigorous and fair peer reviewers and increasing collaborative discussions about papers (YouASTRO tries to go in the latter direction, but I admit results may be far in the future). Actually, it is indeed the increasing number of papers who urged me to realize the webapplication to categorize it and keep track of them in an efficient way, so I must say that I see Anatoly point.

  6. EVESQUE Pierre May 12, 2012 at 4:32 pm

    I do agree with Anatoly. In fact it is not easy to evaluate research.
    I am quite interrested by Fabrizio proposal (http://www.youastro.org).

    But first let me remark the following about evaluation:

    1) One of he main problem with peer reviewing is that there is not so much material to study: What can we analyse?
    The referee reports and the discussions with the editors are considered as usual corrrespondance, which can not be published.
    So we cannot have access to correspondances of other; and the editor cannot do the job too, first because he is one of the party, second because if he detect and declare some flaw, he will attack the notoriety of the journal.

    This is the main problem against scientific understanding and scientific evaluation of peer reviewing. The simple defense is then: if we work without peer reviewing we will read bad papers (I will reply to this argument with practical example later)

    2) I tried to turn this rule, and I succed partly, but after a lot of effort:
    a) I gave as a testimony all my articles which have been rejected, with the correspondance in a booklet duruing a general assembly of my Lab managing meeting. This allows everybody of my colleagues to look at it. It principle also this is accessible to the CNRS and to the evaluating staff of labs.
    Also the document can be asked to the director of my lab, or to myself.

    In such a way, I will be able to discuss of the problems and to quote them through real examples.

    if som of view is interrested, please ask for the document. It is hard to discuss the papers, but it will be the main way to force the change.

    3) This took me about 10 years to do so: It is only after some appeal to the CNRS headquater (which did not answer), and after asking some help to the CNRS Mediator, that I could discuss, using the mediator intrmediacy, with the CNRS law service. They agreed that I could use this protocole. The lab director tried to refuse, but I scceded at last (after 6months).

    This procedure can be used by any one working in a cnrs lab, or directly to the chancellor of the University.

    4) At the moment nobody analysed the documents…. But we can expect it.

    The trick is that peer reviewing expect to restrict the possibility of its evaluation. How can we change this? It is obvious also that changing the role plaid by evaluation may change the habits, and change the measure.

    I will continue later.
    Please also take a look at a journal I transformed 15 years ago to change it on a different way to play peer Reviewing:

    see : Poudres & Grains

    Best regards
    Pierre Evesque
    pierre.evesque@ecp.fr

  7. EVESQUE Pierre, Lab MSSMat, ECP May 14, 2012 at 9:55 am

    About Poudres & Grains (P&G), http://www.poudres-et-grains.ecp.fr/spip.php?rubrique1, :

    I tried to define a journal with correct scientific rules : This one require to be able to debate.
    The scientific debate is not always possible as a normal Peer Reviewing rule, due to the a priori judgment. But it is required for science, as in any PhD presentation.
    This P&G is a journal with open access; it is “limited” to « professional » research scientist, or equivalent, because nobody can assert every thing is true , especially in research field. Anyhow, no journal can assert it too. …
    So, those who are (or consider to be) professional can read, and discuss.
    The reviewing is based on readers, who tell mistakes as they find them.
    So the reviewing is done by the readers themselves, or by the author if he finds some mistake after discussion/or presentation in a conference.
    Of course there is nothing to declare when the readers do not detect mistakes. So no reviewing comment does not mean no review process in the proposed way of a posteriori peer reviewing.
    [In an other way, no possibility to attack the (a priori) peer reviewing does not mean no problem…]

    For instance, most of my papers in P&G (if not all) were presented in public scientific meetings/congress. They pass the scientific discussion; I did not feel any important question.
    It is the same for papers from others. One P&G 15(3),35 has been published few years after without any problem, the others (P&G ns2 &3) have been explained and discussed in conferences as Powders & Grains, and elsewhere….

    What is very strange is that no reader desired to discuss the articles, neither for my own papers nor for other ones (P&G ns2 & ns3, …).
    Better or worse!
    I asked in 2001 (and the years after) section 05 of CNRS to review the papers; it refused.
    I asked also the French Academy of science… No one wants.
    At the beginning I was asking for publishing the report which should be signed by the person. Now I am just requiring for my own papers that the institution (CNRS, editors, …) signs the report.

    The problem is that without report, the Journal can not prove the existence of reviewing process.

    One of my paper was translated to be printed and published in a book (with peer reviewing).
    Few other where sent to normal peer review J. and rejected ; but how can I publish the comments? Please give me ideas and rules that would be receivable by editors….
    I will continue later….

    Pierre E

    PS: Of course I do agree also with Ted….
    It is quite fun: let people talking without reviewing, people do not use.
    On the contrary add a review the sentence is clearly yes, then everobody goes… Why? Is the process better? , I do not feel.

  8. Pierre Evesque, Lab MSSMat, France May 14, 2012 at 5:11 pm

    I am continuing. But I am not sure that we can still face the reality.

    About Poudres & Grains (P&G), http://www.poudres-et-grains.ecp.fr/spip.php?rubrique1, :

    I tried to define a journal with correct scientific rules : This one require to be able to debate.
    The scientific debate is not always possible as a normal Peer Reviewing rule, due to the a priori judgment. But it is required for science, as in any PhD presentation.
    This P&G is a journal with open access; it is “limited” to « professional » research scientist, or equivalent, because nobody can assert every thing is true , especially in research field. Anyhow, no journal can assert it too. …
    So, those who are (or consider to be) professional can read, and discuss.
    The reviewing is based on readers, who tell mistakes as they find them.
    So the reviewing is done by the readers themselves, or by the author if he finds some mistake after discussion/or presentation in a conference.
    Of course there is nothing to declare when the readers do not detect mistakes. So no reviewing comment does not mean no review process in the proposed way of a posteriori peer reviewing.
    [In an other way, no possibility to attack the (a priori) peer reviewing does not mean no problem…]

    For instance, most of my papers in P&G (if not all) were presented in public scientific meetings/congress. They pass the scientific discussion; I did not feel any important question.
    It is the same for papers from others. One P&G 15(3),35 has been published few years after without any problem, the others (P&G ns2 &3) have been explained and discussed in conferences as Powders & Grains, and elsewhere….

    What is very strange is that no reader desired to discuss the articles, neither for my own papers nor for other ones (P&G ns2 & ns3, …).
    Better or worse!
    I asked in 2001 (and the years after) section 05 of CNRS to review the papers; it refused.
    I asked also the French Academy of science… No one wants.
    At the beginning I was asking for publishing the report which should be signed by the person. Now I am just requiring for my own papers that the institution (CNRS, editors, …) signs the report.

    The problem is that without report, the Journal can not prove the existence of reviewing process.

    One of my paper was translated to be printed and published in a book (with peer reviewing).
    Few other where sent to normal peer review J. and rejected ; but how can I publish the comments? Please give me ideas and rules that would be receivable by editors….
    I will continue later….

    Pierre E

    PS: Of course I do agree also with Ted….
    It is quite fun: let people talking without reviewing, people do not use.
    On the contrary add a review the sentence is clearly yes, then everobody does… Why? Is the process better? , I do not feel.

  9. Pierre Evesque, Lab MSSMat, ECP, France May 15, 2012 at 2:46 pm

    15/5/2012
     A) Continuing on the last question of 14/5/2012… (and please excuse me for my English writing)

    There are probably different answers to the “efficiency” of a priory peer reviewing, which I tell:
    1) One can be granted and funded through this review.
    2) One can correct one’s article and get improved vision of oneself.
    3) The errors will not be displayed,
    4) One can even learn how to make sentences…. (after few articles and few referring processes).

    5) While giving results and helping understanding, without any outcome, is completely stupid….

    …. But this (5) is just how science works! And this is why science worker needs ethic and luck. We can help one another to write correctly, but is it really the ethic of science? Yes if, and only if, the guy one helps is at the right level; otherwise the person may become a peer reviewer and the system will fail.
    In the past, article, once rejected, should not be resubmitted…. This seems no more true, and most papers are published after few reviewing processes and few submissions to different journals…

    Is peer reviewed literature easier to read? It is not obvious. One can compare with papers published in Poudres & Grains, or elsewhere; better, as scientific literature is produced by scientific workers mainly, who work in institute… one could ask them to ask for scientific advice prior to submission…. Does it contain less flaws, probably not….

    B) Continuing a posteriori peer reviewing and Poudres & Grains

    This is why I have been preferring trying the technique of a posteriori peer reviewing in 1998, using the possibility of new techniques to amend the papers if necessary.
    Also, at the time when this question happened, I was convinced that my papers which were rejected by editors was not rejected for right reasons (see my testimony n°1 at Conseil Laboratoire MSSMat on 23 June 2011 in the Intranet at http://www.mssmat.ecp.fr/IMG/pdf/Temoig_editions_pieces.pdf , sorry but you need also to get the password from the Director of thhe lab at hachmi.ben-dhia@ecp.fr ) . (Please also wait a little so that I will have time to sum up these correspondences).

    About the validity of Poudres & Grains papers
    First my demand of publishing through Poudres & Grains was not considered as correct by many different research managing committee. These committees never tried to help me fighting against bias peer reviewing, they never tried to force an ethic gentleman agreement.

    ► As I told it already, I asked few times for some a posteriori reviewing to CNRS and lab, editors, ….

    ► I reported these correspondence and acts in TemoigEvesq-CL16Dec11.pdf at the Conseil Laboratoire MSSMat of 16 Decembre 2011 http://www.mssmat.ecp.fr/IMG/pdf/TemoigEvesq-CL16Dec11.pdf sorry but you need also to get the password from the Director of thhe lab at hachmi.ben-dhia@ecp.fr ).

    This is perhaps linked to the field of Physics and mechanics of granular matter, which is an old field from the point of view of engineers but which is a “new field” fo physicists (to whom I belong). This community grews up fast at the end of eighties and I left it for learning more on techniques in the 90-92. Due to this, I add to fight some wrong understanding very rapidly (P&G 7 (1999)) and to introduce new model comling from engineering to the physics community… Please take a look at P&G 19-3 (2011) to understand better.

    ► So as I told it already, (i) most of these papers have been discussed in meetings. (ii) No body wrote to me to discuss any of them, (iii) I published few papers to overpass incompetent peer review processes (unfortunately, there is no possibility to publish private correspondence so that I could not publish the report, but the authorized person can take a look at my testimony n°1, where these are given). (iv) one of it was published later, after English translation, into in a scientific book, with coworkers. (v) one was written to evaluate an NSF proposal. …Recently, I turned to the granular gas problem which I found to be ill treated by theorist (from my point of view). What is fun is that no scientist discuss the problem. They ignore it, as if it was a correct use in science.

    ► This is just what a priori peer reviewing seems to allow. Not to refer means not to discuss, not be be true… This is just ridiculous.
    This is one of the ancient trick used by religious, … to get justified… But in France, scientists from Academy des sciences are only allowed to use this argument for production of work without the use of two temperatures…

     I will continue later , probably with the analysis of some case of article rejection or misconduct

  10. Pierre Evesque, Lab MSSMat , ECP , France May 16, 2012 at 3:57 pm

    16/5/2012 Continuing from 15/5/2012

    – Before coming to the description of few flaws in “a priori Peer reviewing”, let me first stress the difficulty (i) to make these flaws recognized by the editors themselves, (ii) to advertise other scientists of these flaws, (iii) to be able to demonstrate some error in the literature, when this demonstration would need complete discussion .

    As a matter of fact, one can find in scientific literature many cases of good scientists declaring that they could have never published some of their important work since publish before creation of a new Journal editing a new Journal.

    – How to publish referee report?
    An other point is the difficulty to publish referee report: as I told yesterday, a major problem for P&G edition is to demonstrate the validity of the papers, since nobody seems wanting to write scientific comment there. So I decided to send one paper for reviewing to a conference- Journal , editing papers after Peer Reviewing, and to edit the comment. But the law protecting the private correspondence(see 1) is so strong that I had to impose the following tricks. First the paper for which I asked the review was in French; I translated it automatically for the reviewing, then I did not mention any Journal/or/editor/or/referee names; I translated automatically the review from English to French; I introduce some minor differences and published the new text with my answers at P&G 19, 5-11 (2011).
    Little after, I tried to get this comments accessible on ArXiv. ArXiv did not accept because I mention explicitly a paraphrase telling refereeing process or referee report. I will try another way.
    Please let me know any idea.

    *************************************************
    note #1 : see for instance the 2000 judgment at the TGI-17 th correctional court in Paris, France, at http://www.lifl.fr/~ryl/ens/MasterTIIR/documents/CNRS_jurisprudence.pdf . This was for electronic mailing.

    ***********************************************************************

     Part F: Some Flaws of a priori Peer Review edition

    Coming now to the problem of evaluating the efficiency of a priori Peer Review edition I will start with few cases which are not taken from my own articles. I hope this will not stress any person. This is not at all my purpose; the cases are relatively old.

    ►F1 : In 1997-98, I had to review a paper on vibrated granular bed for Phys Rev Let. , by B.Thomas & A.M.Squire (PRL 81,574, (1998). The paper was quite nice; it reported on old (i.e. well before my own work in 1988) experiments of the authors using a fast camera; but the second referee tried to ask too many modifications to reject the paper (in particular asking too many photos). On the Appeal the third referee (PG de Gennes) agreed with my position and accept the paper, writing severe sentence on the second referee comments. In a normal peer reviewing process, such problems should be avoided and should lead to conclusion.
    Did the PRL editors decided not to use anymore the second reviewer? (Let me think no).
    Please Readers, editors, do comment.

    ►F2 : In 994, appeared in Nature (374, 39-41) a very nice paper on segregation in flows in granular matter entitled “Avalanche mixing of granular solids”, by G. Metcalfe, T. Shinbrot, J. J. McCarthy & J. M. Ottino. (I learnt after the group was quite well known for their studies on « chemical mixing » in liquids). After reading, I did not understand why there were using avalanches in a rotating drum. As we planned to study segregation with a PhD student, I asked the question directly to one of the authors, who answer immediately the right information : Mixing in permanent steady flow can happen only in 3d not in 2d, as everybody knows from (i) the parallel between the equations of 2d Hamiltonian problem and of 2d steady flow, and (ii) the law of turbulent mixing and chaotic motion, (iii) writing the equations and mentioning references (probably already included in the paper). This was quite fair, and allowed to strengthen our goal to work in 3d mixing and 3d segregation and to study Turbula….

    I tried to explain the problem in seminars, but Nature is so famous that I was considered as a troublemaker. Nevertheless, ten years after a scientific study demonstrated unambiguously the existence of regions with slow dynamics mixing near the non moving walls of an agitated container (Phy. Rev. Let. 99, 114501, 2007). A result that any cook (or concrete maker) knows: one needs to scrape the cream or the paste near the bowl walls in order to get homogeneous paste fast. Nevertheless the thesis is quite interesting and the PhD work beautiful; and it shows that good science concludes similarly when experiments are carried carefully.
    But ten years to admit these points, “simply” because of a wrong advertising in a famous journal, this is paid quite expensively.

    So the question is, (Please can one give some advice /Comment?) :
    Can one accept that scientific literature does hide recent knowledge in order to make the paper accepted. It is obvious that the referee did not understand the true story: if they did, (i) they should have rejected the paper, and/or (ii) have asked for changes to make the story much more pedagogical.
    It shows also the difficulty of advertising a research domain from another one. The scientists who published Nature (374, 39-41) wanted simply to advertise that they were knowing some more than what appeared in other segregation papers; and try to transfer their knowledge from the “new” domain.

    Please can one tell any advice /Comment? :

    ►F3 : The third case happened in Nature around 1997 on the same topic (segregation in piles). The paper was right except the following assertions: (i) the problem has been known only recently, (ii) no solution was proposed already (these are needed arguments to be published in Nature!).
    I wrote a letter to the editor for misinformation, telling that (i) the problem has been known for 50 years, that (ii) a similar solution was given about 10 years ago, that (iii) referee did not make their jobs, that (iv) misinformation was also an editor problem through the management of referee pool.
    I got no excuse from the editor who proposed to me to rewrite/resend my letter to the author, who was a nice guy. I replied it was not my problem but his own one. Nothing happened.
    More than ten years later, I asked Nature to publish this correspondence on the web. No positive answer.
    Copies of the documents are available in “TemoigEvesq-CL16Dec11.pdf” (see above); they were also included in my 2009-10 CNRS report.

    I will now start discussing my own papers, which have been rejected by editors. I will start with those of my previous field: atomic, molecular or ionic impurities in crystals studied by time resolved spectroscopy at low temperature, and related issues(see note #2).

    ********************************
    note #2 : This was in the time of my second thesis (doctorat d’état) in the early eighties (1980-84). I was interested in studying trapping processes and the transfer to other impurities of an exited states belonging to a given impurities belonging to crystalline solids: how does this proceed? Can it jump from impurity to impurity… The trick consisted in exciting the impurity at some time by a pulsed laser, and to study the time dependence of the different emission lines (i.e. emitted at different frequencies). At that time a debate existed in perdeuterated naphthalene (Nd8) crystals doped with perhydrogenated naphthalene (Nh8) impurities. The studied excited state was a long-living triplet state when the impurities was at low concentration. Increasing the impurity concentration makes the transfer from impurity to impurity possible; then the excited stated states could reach and a lower-energy trap (where it staid trapped), or could find a second excited state, with which it coalesces, generating a singlet state that luminesces at once. The question was to determine whether the excitation could migrate though an Anderson transition (a quantum process) or through a percolation process (a jump process using a tunnel effect in the present case). My experiment has been able to rise the indeterminacy by measuring directly the spectral exponent of the 2d percolation through the time dependence of the singlet state, at the same time that Alexander and Orbach was determining this spectral dimension and that de Gennes helped me to understand the compatibility/relation between our two works.. .
    ******************************

  11. EVESQUE Pierre May 19, 2012 at 5:03 pm

    this is to discuss with Ibrieda and Igedijus:

    And if I well understood the problem of Newton was not to be attacked on his faith. So he only published his “principia” book after he got convinced not to be attacked, from personal colleague.

    I think that many of his handwriting works are not published still!!

    and similar for Darwin!!

    On the countrary, in our system and time Galileo will not be able to save is life by rejecting his book, because it ilooks no more possible in scientific literature.
    Have some comment about this last point.
    It looks as if we can no more reject some thinking, which was thought right as some time, but we wrong now.

    Pierre Evesque

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: