In his day job, Nicholas Holmes does research into hand movement and sensation at Nottingham University in the United Kingdom. When the country went into lockdown during the pandemic, he missed chatting with colleagues.
So he decided to start up a discussion on the internet about something he himself thought was important: the quality of research and science journalism. The result was the website and podcast The Error Bar, which does fact-checks on news about neurology.
“I wanted to talk about science and about mistakes. How the media monitors research has also always interested me.”
Decided to anticipate the reviewer
The podcast had a handful of appreciative listeners, but Nicholas Holmes received much more notice when he focused on his own work. In a series of tweets during spring 2021, he reviewed all his published articles critically, and chose something to comment on in each one of them. Or, more specifically: in all except one. That article he was completely satisfied with.
“What gave me the idea was actually another researcher, who is campaigning hard to make science more open, and started up a kind of ‘openness scrutiny’”.
It investigated earlier publications systematically – how open they had been, and how good the researchers were at sharing their data. Articles were commented on, graded, and a top-ten list began to take shape. “This wasn’t enormously popular, exactly,” establishes Nicholas Holmes.
“I started thinking myself about what could happen if someone reviewed me. So I decided to pre-empt this process by pointing out my own mistakes instead.”
Positive response to self-criticism
The response was mostly favourable. “The best contribution to academia I’ve seen”, “Fun, but a bit scary!”, “Hooray! What an example”, were some of the comments. Nicholas Holmes believes that this is because those who use Twitter are mostly junior researchers, who are positive towards change.
He was also asked to do some interviews, and was invited to write a column in the periodical Nature. He also heard from an editorial office that had published some of his scientific works. He smiles a bit.
“They had seen the tweets, and contacted me to ask if I wanted to make any clarifications about the things that I had published with them. That was good, I think. Perhaps it wasn’t optimal to just contact one or two authors and ask them to do so, but…. still.”
More focus on what is selected for publishing
Nicholas Holmes has always had a curious and critical view of the world. He loved his time as a doctoral student, when he was able to focus on one thing only, and “analyse it to death”. But it was also during this period that he began to realise that not everything was a simple as it seemed.
Scientific ‘truths’ were not always true, and data in the neat experiments were not always stable and reliable. Not all results were even published.
“We can, of course, not publish everything we do; we have to be a bit selective for practical reasons. But in recent years, there has been more focus in the research community on what we actually choose to publish, and how we can scrutinise the processes. Internet and social media have made it possible for anyone to take part in the discussion, and I think that’s brilliant.”
Free and continuous criticism
Nicholas Holmes talks about the open science movement, that ever more people agree that research must become more open to scrutiny. Not just conclusions, but also data and methods need to become accessible to more people.
It is strange, he points out, that so much of what is published is still found behind paywalls, so that students, for example, or researchers in poorer countries, cannot access it.
“This will change in the future. New data, ideas and methods come along so quickly that the author of an article no longer has the last word.”
But surely this is how it has always worked, that new ideas and results overthrow the old?
“Of course. But if you want to criticise something that has been written, by yourself or someone else, then it has been necessary to formally send your comment to a periodical. That takes time, and they can refuse to publish it. I think that it should instead be possible to comment continuously on all published articles.”
Different drivers throughout the career
In his column in Nature, Nicholas Holmes wrote that most self-critical persons are university students. They are much more critical than doctoral students, and further up the career ladder things just get worse, on the whole.
This is actually not particularly odd, he thinks. Many students are not considering a career as a researcher. They just want to get a good degree, and you do this by doing what your teacher thinks is right, which often includes being brutally honest about your own weaknesses.
A person in the middle of their career has completely different drivers, however: What counts is how many articles you publish, which requires exciting data. This makes it tempting to take short-cuts and to exaggerate.
Nicholas Holmes describes an incentive culture that rewards positive results rather than “the messy truth and experiments that don’t work”.
Pre-registration rewards good research
But he thinks that things are moving in the right direction.
“Today, there are more periodicals that accept ambiguous or negative results, and that use pre-registration. Then it is your method, rather than your results, that is accepted for publication. This means that a peer review is based on your research plan, not on whether the results are exciting. In this way, they select for good research – because it is about the process.”
Pre-registration (see fact box) reduces the risk that negative results or zero results remain unpublished and unknown. When studies that disprove the researcher’s own hypothesis, or that fail to produce clear results, remain unpublished, this can become a handicap over time.
It can lead to others testing the same hypothesis at a later stage, or doing a similar trial, which is of course a waste of time and money. Nicholas Holmes thinks that all results should be made accessible in some form.
“We should encourage other ways of publishing than just traditional articles. And we must start rewarding good methods, not just good results.”
Periodicals must demand data
“The old ways of thinking are beginning to change, from the bottom up,” says Nicholas Holmes, and points out the new periodical JOTE, Journal of Trial and Error, founded by a group of doctoral students.
He is also seeing promising initiatives from funding bodies, such as the large British organisation Wellcome Trust, which has reinforced its openness criteria, and from the periodicals themselves. DORA, the Declaration of Research Assessment, is a set of guidelines formulated by the heads of scientific periodicals. The goal is to develop better ways of reviewing research results.
“In actual fact, I like the periodicals’ positions as gatekeepers, that science is reviewed by persons who are qualified in their field. But these are new times. You have to permit more open comments, and you must require researchers to make their data accessible.
Today, however, few periodicals do so,” he establishes.
“Several years can pass before publication, and once it happens and someone asks for data, then they are hidden in a computer at the last workplace, or with a doctoral student who has left. This is not the way to do things.”
Nicholas Holmes stops to think at bit.
“I would like to take this chance to say that I’m not at all negative to research. Of course, I do have a lot of views about what is wrong, but I really love doing research! It’s like solving riddles, or being part of a fascinating, gigantic game. It’s just that I believe that it could be played better.”