Super predators moving at the speed of peer review

There are many good reasons to start a science blog and neurocopiae has collected a few. Science is what you make of it.

Advertisements

Since this blog is going to be all about the scientific search for the truth, I have to come clean first: I have been pressured lately to share my thoughts with you. Many interesting new journals and conferences have reached out to me “because of my eminence in the field” in the past weeks. I have been flattered by the many read requests to emails that permuted my name in order to grab my attention somehow. To be honest, I was quite surprised to reap the benefits of my scientific outreach. Who would have thought that I could join Atomic Physics 2016 as an honorable guest, Neurosurgery 2017 as a speaker, or the International Journal of Medical & Clinical Imaging as an editor?

However, the greatest success of my ambitious attempts to connect with other aspiring fields was certainly the several invitations to speak at engineering conferences such as the Oil & Gas Expo 2016. It was much easier than what I expected. I just had to smuggle the word “fuel” in as part of the title of a recent paper. Now, I can sit back, relax, and watch the invitations fly in. Sweet!

The cost of being in the predatory spotlight

But this sudden “eminence” comes at a cost. I don’t have the time to go to every advertised meeting to learn who the keynote speakers are. At times, I feel bad that I don’t even know any of the guys who are apparently spearheading my field. Furthermore, I feel bad that I don’t have the money to join these exclusive meetings for the special knock off prices (>$2,000 for early birds) offered to eminent researchers just like me. To deal with this new-found burden, I have good news for all of you. I am starting a blog instead. How is that? Mark my words: It is going to be free, and open, and great. Join me and we will skyrocket the science blogosphere!

Other than handling the many requests of predatory journals in my preferred way, there are many reasons to start a science blog these days. Don’t worry; the list will not be exhaustive as I will focus only on the good ones (for now).

Young, talented and fed-up

First of all, there is a lot of frustration with the current state of research, which is being channeled for very good reasons. At Nature, “young, talented and fed-up” researchers tell their stories about the daily struggles of junior researchers. But the frustration is not only about the decline in grant-independent funding and the fierce competition with senior researchers for scarce monetary rewards. The frustration is also about how science used to be done by senior researchers and the painful transition that we are witnessing at the moment. The reproducibility crisis has hit psychology hard and action has to be taken to turn things around sooner rather than later. I can certainly feel the excitement that the revised way of doing science has prospectively sparked in many formerly disillusioned young researchers lately, who thought they must be doing something wrong when they did not get the expected result. It turns out they probably did everything right. We need to build on the inherent promise of change if the transition is supposed to be transformational. Otherwise, we will lose a lot of talent on the way.

However, if we raise the bar to claim a new discovery, as many researchers have advocated in the last months, we risk putting another heavy burden on the shoulders of young faculty members. I know many young scientists who are trying to do things right, but are still judged based on the numbers of published papers and the impact factor of the journals they publish in (disclaimer: I am a junior faculty member as well). Given the same funding for staff, we can’t expect to do better science while keeping up the current rate of papers. I know that the latter is not a good goal anyway, but this is what is still being incentivized by job committees. Whereas I don’t want to spare the surprise of future blog posts on the neuroscience of motivation, I think we can safely agree that institutional incentives play a big role in shaping the career goals of young researchers. Should you run a replication study before you publish a new finding? Most certainly. But what if the replication delays the publication of the results? What if someone less rigorous is faster and you get scooped, reducing the “novelty” of your research, which is at the heart of a high impact publication? What if your replication fails, reducing the potential impact of the resulting paper? And what if rerunning the study takes again 3-4 years as is true for many of the bigger studies conducted in fMRI research (my field)?

Enter the peer review lottery

These open questions strongly add to the second reason for starting a blog. I am very frustrated with the way peer review is currently handled. As an author, you put in a lot of effort to draft a good account of the study you have (finally) completed. Although online data storage is becoming cheaper every year, many journals still have word count limits imposed that force you to relegate important information to the supplement. You may craft a brief summary and sacrifice comprehensibility, but this may leave the next poor grad student in tears in trying to replicate what you have been doing. As an author and a reviewer, I hate the split between the paper and the supporting information just to keep the word count within bounds. If it takes ages to adapt procedures as simple as these, how long will it actually take to change the way we do science for the better?

I know we all are overwhelmed by review requests and we as reviewers have little time to browse through the whole submission to locate necessary, but scattered details. However, I find that when some reviewers decide to ignore information provided in the supporting information, this may turn out to be decisive. Interest in the different parts of the manuscript tends to be widely distributed across the pool of potential peers. For example, when one of my papers was rejected by a prestigious psychiatry journal because of lack of detail on some of the elements of the paper (pointed out by two of three reviewers, the third said it was great), I thought this would be easy to solve. We appealed to the decision and demonstrated that the required information was already contained in the supporting information of the previous submission. Happily, we would restructure the paper to accommodate the insightful suggestions of the reviewers to the fullest extent. Of course, we want to make the submission a better research report.

However, the editors did not think that this was convincing enough to ask the reviewers to reevaluate it. The paper was turned down, we resubmitted to a different journal, and got rejected again. And again. And again. As part of the process, I learnt that the most important section is always the one you had just shortened (with details moved to the supplement) and traded in for more elaboration on a previously “lacking” part. Maybe I should have dropped the additional analysis to back up the main result for the sake of impactful science? This cycle started before I had even heard of preprint servers for papers and I am still caught in the publication cycle for this particular paper. Sigh. There are many other captivating stories left to be told about peer review and I will keep collecting them at conference sessions. The bottom line is: everyone agrees that some sort of peer review is indispensable at this stage. Nevertheless, it is so variable and unpredictable that no one can count on the rapid dissemination of important insights via peer review. At least, we need to make the best out of the scarce time available to reviewers.

Role models of methodological terrorism

My third motivation to start the blog is the many great examples of other science blogs. They are not peer-reviewed, but they are great and I can evaluate them myself. I enjoy reading posts of popular “methodological terrorists” a lot, which is why I will not be dropping names (in honor of the person who coined the term). They are great for instructional purposes or teaching and often much more accessible to students in many regards than (pay-walled) book chapters. If you have made it this far, you might be wondering why I need to set up another blog then. The reason is quite simple. I sincerely believe that we can only transform neuroscience research for the better if we bring the message of methodological terrorism to every village. In this endeavor, peer review as it is at the moment would be slowing down the progress to a degree that appears unbearable to me. For example, in the more applied fields of nutrition, obesity, or psychiatry that I call my scientific homes, there is still a huge disconnect between the promised land of open and reproducible science and today’s common practice. Our research on reinforcement learning shows us how hard it is to break habits and in the end, we are bound by the same fundamental rules of motivated behavior. It will take much effort to settle for a better way of doing science. We always have to keep trying though.

Neurocopiae’s mission statement

In any case, I feel that it is my responsibility as junior faculty to weigh in as much as I can. In fact, it is not a lot more at this stage than starting a blog. In addition to the classical one-on-one mentoring of my students, I will share my thoughts more openly now. Up for grabs and up for discussion. In the next months and years, I will work on making my customized scripts available to everyone. My hope is that we can facilitate reproducibility of research by making software tools that provide additional insights easy and accessible. Simply mapping brain response to voxels does not tell us much about the underlying function. Yet, this aspect is at the heart of common software packages. I feel obliged to do my share in supporting and fueling the movement by at least blogging about the limitations of current approaches to data analysis and how we can do better perhaps. This is not going to be about thought experiments only, but how to put it to everyday practice. The most important thing is that everyone can join the ride of neurocopiae and be part of the wave. From now on, I will keep you posted on what is going on from my perspective. Instant, open, and unfiltered. Much like the spark of an idea before it comes to business. Let’s see if it catches fire.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s