Chris Chambers is professor at Cardiff University where he is Head of Brain Stimulation. He is also one of the pioneers behind Registered Reports, a type of article where researchers receive peer review and in-principle acceptance before the results are known. In this conversation, we focus on Registered Reports and talk about how Chris got Registered Reports started at Cortex, how the review process differs between Registered Reports and regular papers, whether they are suited for scientists on short-term contracts, and what the future holds for Registered Reports and scientific publishing in general.
BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith. In 2022, episodes will appear irregularly, roughly twice per month. You can find the podcast on all podcasting platforms (e.g., Spotify, Apple/Google Podcasts, etc.).
Timestamps
01:51: What are Registered Reports?
07:24: How Chris got Registered Reports started
16:33: Reviewing Registered Reports and regular papers
25:23: Evaluating whether Registered Reports work
28:52: Are Registered Reports feasible on short-term contracts / scheduled reviews
38:50: Peer Community In Registered Reports / authors can choose which journal to publish their Registered Report in
50:25: Do we even need journals?
54:18: Does Chris ever get tired talking about Registered Reports?
Podcast links
Chris's links
Ben's links
References
http://neurochambers.blogspot.com/2012/10/changing-culture-of-scientific.html
Chambers, C. (2019). The seven deadly sins of psychology. Princeton University Press.
Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49(3), 609-610.
Chambers, C. D., & Tzavella, L. (2021). The past, present and future of Registered Reports. Nature human behaviour, 1-14.
https://www.cos.io/initiatives/registered-reports
Hardwicke, T. E., & Ioannidis, J. (2018). Mapping the universe of registered reports. Nature Human Behaviour, 2(11), 793-796.
https://rr.peercommunityin.org/
https://en.wikipedia.org/wiki/Grigori_Perelman#Geometrization_and_Poincar%C3%A9_conjectures
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., ... & Nosek, B. A. (2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour, 5(8), 990-997.
Chris Chambers is professor at Cardiff University where he is Head of Brain Stimulation. He is also one of the pioneers behind Registered Reports, a type of article where researchers receive peer review and in-principle acceptance before the results are known. In this conversation, we focus on Registered Reports and talk about how Chris got Registered Reports started at Cortex, how the review process differs between Registered Reports and regular papers, whether they are suited for scientists on short-term contracts, and what the future holds for Registered Reports and scientific publishing in general.
BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith. In 2022, episodes will appear irregularly, roughly twice per month. You can find the podcast on all podcasting platforms (e.g., Spotify, Apple/Google Podcasts, etc.).
Timestamps
01:51: What are Registered Reports?
07:24: How Chris got Registered Reports started
16:33: Reviewing Registered Reports and regular papers
25:23: Evaluating whether Registered Reports work
28:52: Are Registered Reports feasible on short-term contracts / scheduled reviews
38:50: Peer Community In Registered Reports / authors can choose which journal to publish their Registered Report in
50:25: Do we even need journals?
54:18: Does Chris ever get tired talking about Registered Reports?
Podcast links
Chris's links
Ben's links
References
http://neurochambers.blogspot.com/2012/10/changing-culture-of-scientific.html
Chambers, C. (2019). The seven deadly sins of psychology. Princeton University Press.
Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49(3), 609-610.
Chambers, C. D., & Tzavella, L. (2021). The past, present and future of Registered Reports. Nature human behaviour, 1-14.
https://www.cos.io/initiatives/registered-reports
Hardwicke, T. E., & Ioannidis, J. (2018). Mapping the universe of registered reports. Nature Human Behaviour, 2(11), 793-796.
https://rr.peercommunityin.org/
https://en.wikipedia.org/wiki/Grigori_Perelman#Geometrization_and_Poincar%C3%A9_conjectures
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., ... & Nosek, B. A. (2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour, 5(8), 990-997.
(This is an automated transcript to help with search engine optimization and definitely includes a lot of transcription errors)
Benjamin James Kuper-Smith: [00:00:00] Yeah, by the way I, in preparation, you know, listen to other interviews, he didn't talk so I could find on YouTube. And, um, so listen to your interview on everything hurts with Dan and, um, James.
Chris Chambers: Yeah.
Benjamin James Kuper-Smith: Yeah. And, uh, not to brag, but they said that you were the first Australian guest and you on episode 56, uh, you will be episode 53, 54 for me.
And
Chris Chambers: So.
Benjamin James Kuper-Smith: you will be my fifth Australian.
Chris Chambers: That's so funny. I noticed this funny thing in, uh, just, uh, related to this in a way um, when people would cite my work, like on which the reports in their papers, I was always for ages in the forties, strange. Like if you look down the reference list, I was always like between 40 and 45 in so many papers. And I it's just, of uncanny.
Benjamin James Kuper-Smith: And empirical papers or.
Chris Chambers: Oh, all kinds of papers. I was just like, yep. 43, 44, 47 41. Uh, I'm sure it's [00:01:00] coincidence, but it was just like weird, like it tells you something about the point in a structure of the document that people starting surfing, registered reports. So they have to set the same with everything and then they add it.
And so it comes off to they've cited a few dozen references already,
Benjamin James Kuper-Smith: Yeah, exactly.
Chris Chambers: of like, yeah, that was a funny.
Benjamin James Kuper-Smith: Yeah. Um, anyway, so, I mean, you already mentioned the restaurant ports and that's what we want to talk about today. So I guess to maybe slightly set the scene for the conversation, when talk about registered reports, um, what also talk about your book, have it properly interleaved here and there, because I guess it relates to a lot of the discussion around risk reports or various reports can help with a lot of the problems that you address in your book and you have something random.
Probably gonna come up. Um, yes. I'd like to go fairly deep and register reports because I guess I've read here, read about them for a while. I haven't done one myself, but I've been in period. I have peer reviewed twice for it or [00:02:00] helps if you, um, so I have a little bit of an insight, but yeah, as we mentioned before, we started recording, there are still many people who don't know what a veteran report is.
I think a fair bit of my audience also comes from neuroscience where I think it's maybe slightly less well-known than in psychology would be my guests. So can we maybe just start by you providing it in a two to three minutes summary or however long you need,
Chris Chambers: Sure.
Benjamin James Kuper-Smith: a restaurant, what is, what is good for.
Maybe what the differences between that and a period illustration, such people don't get confused.
Chris Chambers: we're all familiar, hopefully with the standard white publishing works and peer review work. So the regular article truck that, uh, authors, uh, uh, familiar. Involves you completing a piece of empirical research and then once it's finished and you've written your entire paper, then you submit it to a journal and it's peer reviews.
Okay. And reviewers look at various different aspects of the work and the editor looks at it. They look at the, rationale, the methodology. But they're also looking at the results. Okay. And they're making a judgment in [00:03:00] many cases, in most cases of how great an impact those results are to the field, how important to those findings are novel, are they, and this introduces a significant risk of bias into the peer review and publishing process because the decision to accept or reject a paper through that regular track, isn't based solely on the quality of the work you're doing the rigor and robustness of the methodology.
It's also based upon the. And this introduces an existential risk for many areas of science. Because you start making judgements about what studies go into the literature, According to their results. Then you run the risk of only publishing studies that produce attractive results or nice results or significant results, positive results, so novel results.
And so we can, we risk fooling ourselves into chasing shadows. Seeing, seeing things that are not there are registered reports were brought in as a way of addressing this problem, because learned over the last 10 years. The bias, the publication bias the [00:04:00] reporting bias that we see in the regular literature driven by this incentive structure for authors to produce beautiful results, to a reproducibility and lack of transparency in science and registered reports were brought in about 10 years ago as a way of addressing this problem at source by splitting the review process into two parts for. The review is just look at your design plan before you've done the research. So your rationale, your theory, your methodology, all of the details, all of the nitty gritty parts that go into planning a really good study or set of studies, and they perform an in-depth review on that. And so does the editor, and it passes muster at that stage, usually after revision. Uh, the article is awarded what we call an in principle acceptance, which guarantees the final paper will be published regardless of the outcome provided the authors follow their protocol. Okay. And that their interpretation is evidence bound, which is to say that it's, it's consistent with the data that they've they've [00:05:00] reported. And what this does is it eliminates outcome bias and publication bias because the decision to accept or reject to paper made before the results are not. Therefore it's not possible for the results of the research to influence whether or not the article is accepted or not. And this has, I think, a great, you know, when we introduced this back in 2013 at cortex, It had a great deal of promise.
There was already a lot of discussion about the benefits of preregistration as a tool for reducing bias and registered reports take pre-registration and they kind of put it on steroids because we with normal pre-registration you, you write a protocol before you do your research. You post it into a public repository.
It's not peer reviewed. Usually it just sits there. There's no guarantee that a journal will publish your article regardless of outcome. It's just there and it has some benefit. But it's only part of the picture with a registered report. You take that process and then you add to that peer review and in principle acceptance, [00:06:00] regardless of outcome. so what we're doing is we're galvanizing the entire scientific process to make it more rigorous, more bias control. Um, and we're already seeing the benefits. So, you know, some of the perimeter. Metta science has come out, shows that registered reports, more likely to produce negative outcomes.
So, so we're more likely to find out that we're wrong in our predictions. When we remove all of the biases that lead us down the garden path, that peer reviewers who performed blinded judgments of registered reports versus regular audio. tend to write those, the registered reports higher on virtually every category of scientific evaluation. We find that they're cited about the same. And we find that they're, they tend to be more computationally, reproducible, which is to say that the actual data that's archived along with the papers tends to be a more reproducible, that the outcomes tend to stem more directly from the data. So, yeah, so far so good.
It looks like the initiative is working as we intended. And so, you know, as we go forward from [00:07:00] here, we need to be thinking. How can we make this better? What are the, are there any, you know, unintended negative consequences? can we make it more accessible and more open and even more rigorous than it?
Benjamin James Kuper-Smith: Thanks. Um, I think you also mentioned, you know, a lot of things that I'd like to talk about later, for example, like how we can improve it, what some potential problems are or this kind of stuff. Before we get into the details of retro reports themselves, I'd like to take a slide. Biographical detour and ask kind of like how you got into the whole open science movement.
I mean, that was kind of my question. And then I, I think I kind of read. The origin story of Chris chambers, uh, working five in science in your book, uh, you wrote what my artist experience with publication bias came in 1999 with the rejection of my first scientific paper, the results only moderately interesting sites and anonymous reviewer.
The methods are solid, but the findings are not very important. Set. Another, that sounds like you were from your very first [00:08:00] article. You are looking for something like a of report.
Chris Chambers: Well, I used to think that, um, I think probably like a lot of junior scholars, I went into science thinking that reports philosophy held across science. I, it would never have occurred to me at that point in my career that we would even need this. I assumed that, um, the way science works. Is that you would design a study, run it robustly. the actual outcomes you get would never be the basis of a decision by a journal to publish or not because to do so, just seem completely around. Do you want to know what the truth is or do you want to know sexy. And do you really want the scientific record of your only full of findings that some reviewer considers to be interesting? Is that how we can to land probes on other planets? Is that how we're going to solve climate change? Is that how we're going to invent new drugs to cure diseases? I don't think so. And so that was a shock. And I was working of course, in a relatively [00:09:00] trivial area, you know, that time of psychophysics compared to, you know, those big questions, I still felt somewhat betrayed, I think, by the academic system.
And it's, um, it very quickly cured me of my nightmare. And so, um, for the next, you know, 10 years played the game as we're all taught to play it. This was before the open science kind of revolution, I suppose, had really taken hold in my area at least. And so I, uh, we, I did what I needed to do, you know, I, I played the game, played it pretty well, frankly.
And was only later in my career, I suppose, about, you know, 10 years, 15 years later that I realized. That I was just as idealistic, actually underneath as I was originally. a kind of being a kind of being just, I dunno, following the rules, but I never really believed in them. And so, uh, what triggered, accidentally fell into all this open science stuff just through foster. [00:10:00] Um, having rejected again and again, because of no results or complicated findings findings that didn't lend themselves to easy interpretations rigorous research that we'd done. And I felt like were other people talking about it as well. It wasn't just me shouting into a vacuum. And so I thought, you know, let's do something about this. Um, and as I wrote in the book, I got a paper rejected from, uh, a specialist journal in my area because of no findings. And, uh, I wrote a very short blog post. I just started blogging back in those days. It was 2012. was a thing, and I wrote this blog post, just saying how I was just had enough of this. This is not the way it should be. And I've, I've, we've all been living under the yoke of the system for too long and an editor named Sergio Della Sala. Who's the editor chief editor of cortex wrote to me and said, would you like to come and join our editorial board? Because this is exactly the kind of philosophy I would like to promote. That we should be judging science based [00:11:00] upon quality, not based on And then that's where registered reports came from. And then were launched a cortex in 2013 at the same time as a couple of other journals. Also independently had the same idea. It's very interesting. Actually. I think that how often in science, good ideas come from different people at the same time, like, there's a kind of undercurrent that pushes us all in a direction.
And then we all come together. And so this format then developed from there. since then has been steadily growing. It's been adopted by, I think around 350 journals now is now, um, as we might discuss later is now moving to the supra journal level. So it's going to be on journals as well. So I think it's very exciting.
Benjamin James Kuper-Smith: Yeah, definitely. Um, just to mention it, maybe for people who are new to the podcast, I'll put references and links to. Anything, we mentioned like talks or papers and that kind of stuff in the description of the podcast, a student have to look for them. Yeah, that's really interesting. I [00:12:00] didn't realize that this started with a blog post.
What I kind of assumed was that you, you know, as you said, you kind of had to play the game and you, uh, I assumed you were like accepted in the field. You'd become a, some sort of editor at cortex. And then you said like, Hey, can we do this thing? And then. I don't know, you read an application to the cortex board of editors other than how this works, um, that then you publish the blog post kind of describing what you wanted to do.
Um, is that common at all that editors reach out to academics who publish a blog post or did you know each other or other than it seems really.
Chris Chambers: we did. Yeah, it's a good, not common, but we, I did know Sergio from, uh, from previous interactions we'd had in Austria. If you, my, supervisor out there, uh, and what I came to learn about Sergio, he said, he's, he's very idealistic as well. He wants to, he wants to try and repair numerous psychology and neuroscience and cognitive neuroscience as well, and this way, and, uh, you know, some kind of corrective measures needed.
[00:13:00] And so, you know, in the last 10 years, we've really lifted cortex from, from a very traditional, uh, neuro-psychology journal into one that is. Has a very similar thematic focus, but is now very much focused on open science as well. So we've really.
led the way I think in the field has moved as well. But, Uh, you know, there was that blunt one blog post I wrote which, which prompted.
He says you had to contact me. then there was another one I wrote just after joining the cortex editorial. Proposing this idea of a registered report that we could do, we could launch this article type in doing so I think, you know, we could try it and see what happens and maybe, uh, we could do something good for the field.
And I, I made the decision to publish that lesser as an openness, rather than just making it a behind the scenes kind of, uh, request because I felt like they needed to be some feedback from the committee. About this initiative before launching it, I wanted to get peer review essentially a lots of people.
And I got it when we've got [00:14:00] hundreds of people commenting on it and making it better. But I also wanted the journal to be accountable for the decision that it made one way or the other journals have a, an institutional responsible. To the community to, uh, do what's best for the community. Not what's necessarily best for themselves. And so I think it's important regardless of the personal relationships we have between editors not to get too cliquey and to always that we put that public mission first. And so I felt that, uh, you know, it needed to be, it needed to be an open letter, which was controversial at the time. But as history shows, it was probably the right decision.
Benjamin James Kuper-Smith: Uh, just because you mentioned you think it should be less cliquey and that kind of stuff you are now editor or in editorial boards. I don't exactly know what all the terms are precisely of several journals. Now. How do you kind of ensure that, uh, yeah, it doesn't become too cliquey and all that kind of stuff you just mentioned.
Chris Chambers: Well, I think, you know, I say by, by making [00:15:00] sure you send to the public mission, so putting transparency at the top of the. This is this number of ways he's doing this. First of all, you know, I think media is very good for, for, uh, reaching people, making clear the policies that you're implementing, making clear why you're doing things, seeking feedback and discussion the community.
As you do things, trying to avoid making too many decisions behind closed doors, because I think that can be, uh, that can reinforce and, you know, power structures, which don't perform very well exclude a lot of people. From decisions that affect their lives and the careers and the scientific record, but also by thinking beyond journals, you know?
And so, you know, I've spent the last five years surrounding myself with people who are smarter than me and are very good different parts of this process. And so we work collectively, I work with a lot of different people on lots of different things and moving beyond journals and seeing the registered reports process as something that, the [00:16:00] philosophy transcends any individual journal, I think has also been a.
Benjamin James Kuper-Smith: Yeah, I think that's one of the most interesting parts that I really want to talk about, I guess that would probably have to talk about reservoir a bit more before that, but that's definitely, um, I think I saw a talk of use that. Uh, you K R I O, or something like that. There's a, there's a video on YouTube.
And you mentioned that there. And I was like, oh, this is, this is really cool. So yeah, I definitely want to talk more about that later. So as I, you know, I mean, I invited you to talk about registered reports because in principle I'm a very big proponent of the idea. It seems, you know, as you said, like common sense and what you should do is.
But as I said, I've also done peer review and been involved with it twice. You know, that obviously some problems that you address in your talks and papers and that kind of stuff. And I just wanted to briefly talk about kind of the extra load involved for peer reviewers, because, I mean, I don't know the one I did or I helped out with it was [00:17:00] always helping supervisors.
One of them was fairly straightforward. They kind of done the study and then they was the. I should say they use that as a very good pilot study and then did it kind of reduced the kind of wants to reduce the space of questions they could ask and all that kind of stuff. Um, and then kind of did the second thing on that that was pretty straightforward to review and it almost felt like why are you can almost just do it?
Like, this is a perfect study, good luck with it. Um, another that I help out with was a lot of work and it came back a few times. And at some point, I just, when I got this thing again, um,
Chris Chambers: um,
Benjamin James Kuper-Smith: and, um, yeah. How can you make, I guess like the, the, the initial problem is that you, as a peer review for a single paper, you have to do it multiple times.
Right. Whereas otherwise you would, I guess, typically get at once and review it. So I'm just curious how kind of, can you make that process more efficient? That it doesn't become. Even if it's better in the long run that [00:18:00] in the short run, you go like, oh, God's a registered report. Peer review. This is going to be a lot of work.
I can't be bothered.
Chris Chambers: But it's interesting. I think, because in theory, there's nothing special about reviewing your registered report compared to reviewing the front end of a regular Piper. And I think you, in a way, I think it shows us one of the problems with regular periods. But if you get a paper with problems in it's designed, but it's got results in discussion, why would you consider that to be any less work than a registered report with the same problematic design, but without the results, in both cases, you could say the problem here is the design. And I think there are two reasons. One is with the, with the regular paper, we get distracted by the results. And maybe because it's all done and dusted, we're more likely to just hit the reject button. If it's floor or potentially, if some reviewers don't like the findings, it's a quicker road to the bin with a [00:19:00] regular paper, if it's all done and dusted.
And so it feels easier. incredibly wasteful because all that work for nothing. If it's going to get rejected again. And also it's probably going to go back into the peer review system again. So it gets rejected at one journal the authors go, well, we've got to publish this paper. So they send it somewhere else.
How many reviewers end up seeing it in the end? 15, 20? It's just, the work is spread out amongst lots of people. Now with the registered report, you're getting it sooner. You're still doing the same evaluation in theory of design and theory and everything else, but now you haven't got the results. So maybe you feel as a review of like, oh, I have a bigger responsibility here. can't just hit reject. Well, maybe. But maybe you feel like you shouldn't, maybe you feel like, you know, I've got an ability to be constructive here and to help them. And so it makes me keep coming back. What you can do though, is as a reviewer, you can always say I've had enough, I've done my bit.
I've helped them enough. And that's as much as I'm prepared to do, and you can leave it to the editor to make a decision that reviewer should never feel compelled to keep coming back and back and back and back [00:20:00] I think poor quality editing for an Edison to continually go back again and again, and a denser reviewer. one of the skills of editing and it's a difficult one, it's a risky one as well in some places. But of the important skills of women to know when you burdened your reviewers enough and to say it right now that the can is on me now. And I have to decide, is this paper going to make it or not? And in theory, a registered report is more efficient for that reason. You can fix problems before they arise. Uh, if you do, if I do, as an editor, reject a paper, potentially saved the authors, quite a lot of work. That would be for nothing anyway, because I would go through, if they did the study without going through the registered reports process, they, they might encounter the exact same objections a year later, and the whole stuff, the whole thing goes in the bin.
Some junior scholars career goes down the toilet with it. Potentially it's much more efficient. I wonder, know, what it says to us. What does it tell us about the [00:21:00] operation of the regular peer review system for a registered reports review process to feel like more work? I think it probably tells us more that the regular review process not working properly more than the registered reports review process. something wrong with it.
Benjamin James Kuper-Smith: Yeah, I think one, what kind of reason for why I think is, can be at least for a single paper, more work is, as you mentioned, you kind of have a bit more responsibility because sometimes you go, this is not good. This is not good. You just mentioned it, but I feel like in a register for, do you feel like there's a bit of an obligation to say like, well, you could do this, you could do this.
Whereas, um, you know, often there can be, I don't know, not necessary. Well, sometimes even fairly minor tweaks, right. That can actually make the paper much better, but it did feel like, yeah, it feels almost more collaborative to me than with other ones, because you go like, okay, there's something here in this paper, but we have to help the people actually make a good, rather than telling them that your idea is good, but [00:22:00] you know, do another.
Chris Chambers: And this is right, and this is actually for me, I think one of the strengths of the format, the peer review process is unnecessarily adversarial for regular papers. It's like a, it's like a trial by fire. You know, you've done your work and you your results are clean enough. And you hope that the reviewers don't notice any flaws in your design and you run the Goldman and you hope. And I think, you know, a lot of the problems that we have in science, uh, stem from of the tactics that authors employ to get through that process. And inefficient and it's it's, it gets nasty and it's, you know, I don't like it. In fact, you know, since editing registered reports, I disliked. Going back and editing regular papers because like, you know this again,
Benjamin James Kuper-Smith: Yeah.
Chris Chambers: whereas when I'm editing a registered report or even reviewing one or even watching one, like I'm entering into a much more productive dialogue with experts in my field who, if I'm an editor or advising on a paper, there's no obligation on them [00:23:00] to do all the work for the authors. can draw the line. And say, okay, I'm going to make some suggestions. I'm going to spend however much time did I spend on peer review regularly. I'm going to spend that amount of time and that's it. That's perfectly fine. within that constraint, which I think every reviewer has the right to apply to that life and to their career and their work, you know, there's a huge benefit for a constructive dialogue. seen amazing example. Of reviewers, literally saving studies through simple comments, picking up nuances and subtleties and complexities in designs, coming up with control conditions that authors never thought of. There's a huge untapped resource there that gets missed in regular review. It's all about getting the paper through, all of the biases that authors are using to get their results significant.
All of the, all of the subjective judgements from editors about what's novel and all this sort of crap, it gets in the way of the real job, which is evaluating the quality of the science. And I think registered [00:24:00] reports reviewed does that.
Benjamin James Kuper-Smith: Yeah, I agree. I guess. Yeah. I never thought about it that way, but I guess in a, in a regular review, It does sometimes feel like you find trying to find the flaws, whereas in register reports, you're trying to find them and then help them as much will supportive.
Chris Chambers: It's a, it's a gift to authors to be able to solve problems before it's too late. can't tell you how many times I've had authors write to me after getting stage one in principle. And saying we really value. This is the first time we've done this. And what we found really valuable was being able to fix this issue.
And this issue before we run the study and review is also kept right and say, I was, I really enjoy being part of a process where. I could actually assist constructively rather than doing a post-mortem on a study that was flawed from the beginning, uh, and statisticians as well. You know, also enjoy being part of this process from the, from the [00:25:00] beginning. so much more value to it. And it's, as you said, it's more collaborative. This can present certain challenges when reviewers and authors want to work together because you do need to maintain a certain independence between those roles in order for the governance of the whole peer review process. To be robust, but there Are solutions to that as well that we've employed. But overall, I think it's, I think it's just a superior form of review. And forward to the Metta science that will test that hypothesis more definitively because at the moment it's all based on instinct and what I've seen, but that's not really evidence.
We need to see evidence that it actually does work.
Benjamin James Kuper-Smith: Are you running some of those studies while you saying I'm giving that to other people?
Chris Chambers: I'm not running them myself. I struggled for the time to do that kind of in-depth medicine because I'm spending so much time on the initiative itself. And also I think there's value honestly, in that meta science being done by people?
other than me, like I often feel I have, uh, I have my own biases about registered reports.
I think they, I think they're [00:26:00] great. And for me to do medicine itself, I always feel there's a risk. I could fool myself in a way into seeing benefits that may not hold up. So I think it's better that researchers who are less invested in all this do that medicines, Hilda bassoon writes very about the importance of separating out the advocates from the, from the, uh, from the people. Test the effectiveness of an intervention. If you start blending those, then it doesn't go in very good direction. So I, I, the, the data is all out. Then there's many journals now that have open review they publish registered reports and regular articles. And so there's plenty of opportunity for Mehta sciences to. Collect all of those peer reviews and do analyses often do content analysis, thematic analysis that could have reviewers review the reviews. There's all sorts of opportunities in which we can start asking that question. So I'm hoping somebody will.
Benjamin James Kuper-Smith: Yeah. Yeah. I guess it probably is better to have someone else do it because if I see a paper by Chris chambers [00:27:00] saying, registrable, it's like, right. It's like, well,
Chris Chambers: He would say that
Benjamin James Kuper-Smith: exactly.
Chris Chambers: I, you know, student that we've got to be careful about not lionizing individuals or, you know, being very careful with, with putting anyone on a pedestal here for an initiative to survive in the. It's got to be bigger than any one person or any one small group of people.
If we're trying to carve out future for silence, it needs to be something that we all do together and every great reform in scientific history that has always succeeded on the basis of being that many people and believe in and demonstrate the effect of itself. I think it's very important that we get. A strong element of independent medicines going, cause it's much more convincing if somebody, other than me says, Hey, Russia courts work. Um, and you know, that's if somebody comes in and says, these are the problems as Tom Hardwick has done, and others have done that, these are the [00:28:00] imperfections and these are the issues that need to be solved. That's really, I find that extremely valuable. that prompts me to go, right? What can I, what leavers can I put? To improve the policies those problems. And so it creates an opportunity for a sort of feedback and dialogue between researchers to try and fix the system as we go.
Benjamin James Kuper-Smith: Yeah, I guess the most convincing thing is if you can satisfy the critics, right? The people who actually are kind of against it, and you can say, well, I've solved.
Chris Chambers: Exactly. There's nothing more, to be honest, there's nothing more satisfying having somebody. This has happened a few times who was deeply critical of registered reports at the beginning, come back to me five or 10 years later and say, you know what? I really liked this initiative. Or you see, they don't even tell you, they just start submitting registered reports.
And you're like, yes.
Benjamin James Kuper-Smith: Gotcha. Yeah. Uh, so like another, I guess, common, um, Potential critique is that it, you know, it takes more time upfront and that kind of stuff. And is it feasible for people [00:29:00] on short contracts, that kind of stuff. Right. And I thought maybe rather than using the abstract, we could maybe use an example of a project that I'm thinking about.
Um, and kind of worked through a concrete example there. So I've got one year left on my contract, um, until my, when I have to finish my PhD and I have one project where. I've already done a few studies and found an effect. I think that's fairly robust and interesting. And. Kind of there's. I definitely want to do some follow-up studies, but the question is, you know, do I do them as separate papers?
Uh, all that's one possibility I'm thinking about right now, should I maybe use the initial few studies as a kind of though exactly what you would call this? Um, but for my stage one submission that said, look, we've done all of this stuff here. Uh, there's still a few questions here. Yeah. We want to number one replicated and answer those questions.
And number two, generalize the findings that we found there, because we found them in a specific [00:30:00] context and we want to generalize them in a way. I think Reggie ball would be a really nice format for that, especially because, you know, if we run a second study, we'd be nice to get constructive feedback about potential errors we made in the first version.
So we don't repeat them and that kind of stuff. But one, you know, big question I have is, you know, I've got. This is a behavioral study. So it's not the most complicated analysis. So it's not, you know, not suggesting doing like a clinical FMRI study or something, but, um, I don't know. How should I think about deciding whether I have time to do this as a retro report or not?
Chris Chambers: Hmm, a good question. And it's a very, um, common scenario. I hear from a lot of early career researchers that on short term contracts. They want to use the initiative, but they're not sure if they've got time. So, you know, you can assume that if you submit a complete manuscript for stage one review, it'll take a few months, at least to go through the whole process to get in principle accepted. And this is a problem. Okay. This is one of the, this is perhaps the [00:31:00] biggest limitation of the initiative. one can argue and I have argued this and over again you save that time at the end, I guess you probably do, because you're more likely to get accepted at the first journal you go to. And then once you finish the research, it's a must see a much quicker path to getting it fully accepted. So that, however, it doesn't change the fact that you do have to invest that time with. And there's no guarantee that you'll get stage one IPA in principle acceptance. So, you know, you could spend months and still end up with, with no with no guaranteed paper in that respect.
It's no different to regular peer review. just that it's happening at a point in time there's you haven't done the research yet, so it slows you down. So there's a few options. One option is that like you've described you, you submit a state on manuscript that has your initial experiments in it, where you lay out your experiment, 1, 2, 3, whatever it is.
And then you propose a definitive experiments to seal. There [00:32:00] might be some overarching hypothesis that emerges from the previous work or some theory you can test and you can submit that and can, if it's sales through, then you can publish it all in one big Rachel report at the end. you know, you can, as you, as you mentioned in the beginning, you can have preliminary studies within a registered report It's fine. Now, of course, you still end up going through this stage one review process, which can take. This is why, uh, when we launched the Pia community and registered poets, which is a super journal platform for peer reviewing one and stage two preprints we introduced a track code scheduled review. The idea here is that with scheduled review, you plan the review process in advance at, uh, during a narrow range of dates in the future before you've even written your stage one manuscript, um, What the editors do to what we call them. Recommenders appear community is they up the review as an advance just as they [00:33:00] would with a regular patient, but they're doing it in advance.
Right? So to review as a booked into review between mistake, like maybe a five day window the future, this saves a great deal of time of one of the, one of the things that people don't realize about that peer review process is that one of the major delays with regular peer review, isn't the amount of time. That review is typed to do reviews. the amount of time that editors take to find reviewers. know, review process is creaking review as are under a huge burden. Most of us declined review invitations just because of workload. And so it takes time. You have to ask a lot of people to get enough reviews. you do this in advance, you save a great deal of time because by the time that stage one manuscript is ready in say six weeks as a ready to go. You're ready to go. Bang. Reviews of back in a week or two, and it's a much more rapid process from there. Our first shedrow or truck submission that PCI registered reports took five days, [00:34:00] five days to go from initial submission to interim stage one decision. because anything about the stage one review process was light not?
thorough. It was just as detailed and. And the reviewers spent just as much time as they would have normally looking at the paper. just that they did it during a planned window. Okay. Rather than it just being so reactive, you know, we're used to getting these review invitations and like, oh, can I do this?
Okay. Hit accept. And then I haven't really planned this. Isn't something I factored into my diary. So I'll do it when I, when some free time comes up, review is a late. Because they're busy. It's, it's an inefficient way of organizing the review process. It's much better in my view to share it. And then people stick it in their diary, they ring fence it, and then they spend that time on it.
And the bank, your decision comes back. And so far that's worked really well. And we've had a ton of such schedule trucks, submissions, almost all of them from early career researchers on short contracts because they, they, they want to try and parallelize the [00:35:00] process. write the stage one manuscript at the same time as the review. Um, I'm being found and you might ask, well, how does the review is know what they're going to get? And this is by, uh, asking authors to submit a one page proformer at the very beginning, doesn't take much time to prepare. It's just like, you know, uh, rationale method, brief description of methods hypothesis, brief description of analysis.
You're going to. Something that can be smashed up in an afternoon, or if you've done some careful thinking about the design even quicker, then used by PCR, registered reports find reviewers. So reviewers look at that. They don't review it itself. That is use it as a guide, like is this in my domain, et cetera. And so based upon that, we line the reviewers up. So that's the, that's the big way you can save time at stage one. If you want a short term.
Benjamin James Kuper-Smith: Yeah, that that was one of the. If the PCI registered reports, uh, that exactly what you call us [00:36:00] approach. That was one of the kind of two main things. I really liked the other being then the, that you can choose a journal if you get amongst some of them. Yeah. If, if, um, you get the in principle acceptance, but, um, one thing I'm wondering about is.
How easy is it to estimate how long it's going to take it? You're right. That stage one report, especially if you know how to carry a researcher who maybe hasn't written a paper yet, or only like one or something, because I've noticed that some of my papers have taken slightly longer than expected to write.
And yeah. So would you already basically have written stage one largely, and then you write the thing and. Yeah. Do you know what I mean? Rather than, because it also seems weird if I say this thing I'll be finished in six weeks and then I'm not. And then they said like, Hey, so where's the paper.
Chris Chambers: Accurate planning is key. If authors are late, then the whole process collapses and it hasn't happened yet. But, um, I think, I mean, this is, this is where some self knowledge is important and good supervision is important for junior researchers as well. So, you know, you [00:37:00] need a good supervisor who is able to assist and guide and, that you're realistic about your time that you've got available.
And also the challenges that you face in, in writing, particularly if you're doing it for the first time, know, there are challenges, but we provide detailed guidance. On how to write a registered report. There's so much out there now. peer community in registered reports, guidelines are very detailed. There's plenty of example, registered reports available across many fields. Now there's hundreds and hundreds and hundreds of published registry reports that can be used authors as a. So, and with that knowledge, a good supervisor and some knowledge about your own workload and challenges you face as an individual in terms of writing and so on, it comes down to kiss careful planning. And I don't actually know of the set schedule tracks submissions with. I don't know how much of the manuscript has been written at the point that they submit their initial snapshot, their initial proforma. I just leave that for them to decide their [00:38:00] own And over time, we'll learn as to how good people are at planning this, you know, um, I suspect that we'll be self-selecting to some extent.
I think researchers will be more likely to use this track they're more confident that they're able to meet their own deadline and you can choose that. Peer community in registered reports. Doesn't say it must be in, in six weeks. What we say is you nominate a date in the future. That is at least six weeks from now. It can't be any earlier in time than that. So it has to be at least six weeks from now, but it could be eight weeks. It could be 10 weeks. It could be longer if you want. we will work to that timeline on our end, in terms of lining up reviewers. And so you it's up to you to decide. What you believe is feasible and realistic on your end.
Benjamin James Kuper-Smith: Before I get to the kind of you being able to choose the journal part. What is this PC? I think I haven't quite understood. Um, I hadn't heard of it before. Um, and I'm not entirely sure what it is because it, [00:39:00] the retro report parts is one part of it. Right.
Chris Chambers: Right.
So all familiar with peer review and it's normally something that is, uh, undertaken by journals and journals usually owned by publishers. Therefore standard traditional peer review process is undertaken under the management of publishers, but it's all done by the academic community pretty much and most, uh, journal editors from the Senator. Professional editors that run certain journals. journal editors themselves are also academics. So the vast majority of peer review. Managed and conducted by the academic community, though it's run through journals and publishers, what the peer community in initiative does broadly speaking, beyond registered reports, just the broader initiative it says there's no reason why all, any of this has to happen through publishers. doing it through publishers where it kind of, uh, surrendering a lot of countries. Over, uh, our own destiny because we're giving them [00:40:00] something they consider to be added value at no cost. And then they're charging us for it later, in terms of accessing our. So it doesn't make a lot of sense to just give that up.
Has charities or publishers who are pulling massive profits. So what peer community in does is it says we're going to do the it's a completely free, open transparent process where it all peer review of preprints. So outside journals, beyond journals, above and beyond journals, performs peer review and those reviews are made openly available on the peer community in way. Whichever particular community. It is. There's lots of them. And then there's an opportunity for authors to take the. That reviewed pre-print journal and, for regular PCR communities, there's journals, which will accept further revisions. They'll just accept those reviews and journals can choose different levels of adoption [00:41:00] and so on. But what it does is it takes back control of the entire peer review process. Um, and it, it basically recognizes that peer review is something that the academic community has always been doing. We've always done this ourselves, so there's no reason to give up. To the corporate publishing world. Um, and we should be doing this ourselves and we should be making publishers actually earn their profits.
We shouldn't be donating them in the way. Peer community and registered reports is one of these communities. Okay. So there are lots of there's about 14 or 15 different PCAs, and most of them are discipline specific. So you'll have like a, in, um, evolutionary biology or a PCI in paleontology. most of them you can think of as kind of discipline, specific layers, peer community in registered reports, cuts through all of those. 'cause it's not discipline specific it's format specific. it's a community dedicated to the registered reports format. so you can think of it as a vertical column kind of cuts [00:42:00] through all of the scientific disciplines non-scientific disciplines. In fact, registered reports is, is open to scholars in all areas of research, the humanities.
And so what PCI registered reports. It's similar in philosophy to what the other PCI is. Do receive so authors post that pre-print on a preprint server, or maybe they as an embargoed, document on the open science framework. We don't actually, we're not a journal. don't publish separately.
We don't we're not, we're not a journalist such what we do is we receive the preprints we peer reviewed. And then we publish what we call recommendations based upon those peer reviews and revisions. So it goes through the same rigorous review process as a, as a registered report, we'll do at a journal. just that there is no journal as such instead, a child is get a stage One in principle, acceptance from us, they can then take that, uh, pre [00:43:00] accepted. And they can take it to a journal if they want to, or they can just leave it as a pre-print. If they wish it's peer reviewed it's up to them. they want to take it to a journal, they have other options.
Now we've got a fleet of what we call PCI, registered reports, friendly journals. These are journals, which, um, guaranteed to publish any two registered report that was, uh, received a positive recommendation from PCI registered reports and they make this. Without requiring further scientific peer review, which is a really big step.
So our review process is replacing the journals peer review process. So that means that authors are in a position to choose from this fleet. PCI offering the journals. They can choose which journal they want to go to, which puts the author in the driver's seat longer are you going from one journal to the next, hoping that the journal will publish your article. Instead, you get your positive recommendation from PCI registered reports, then from the list [00:44:00] of eligible journals, you can choose whichever one. Later at the end, at the end of stage two, you can choose all of their eligible journals will offer IPA when we do, and then you can choose. You can also just choose any other journal, right?
So you don't have to go with those journals. You can submit to any journal you like, they may want to do extra review or not. but you can, you have that power as well. authors much more control over their destiny to be able to take the review process away from publishers and do it ourselves. And then. Give it back in a controlled fashion. I met her that we control that's the strength of it. It's um, it opens everything up. The reviews are open. They can be signed to anonymous, but they're always published. It gives authors control over the outcomes of their work. So where they want to publish it now. And it helps it'll help us to transition away from this kind of dependence that we have on corporate publishers, especially in the long [00:45:00] run, because, you know, we, if we control peer review, then we control publishing in science and looking at.
Benjamin James Kuper-Smith: One thing that. Slightly confused by all. I wonder whether it's going to work is so when it comes to the, basically the, the, you know, so, you know, have, how many is it, 30 in journals or something like that? That's a part of this. Um,
Chris Chambers: Yeah, this, I think there's about 15 different communities. Yeah. And they're growing all the
Benjamin James Kuper-Smith: I just, sorry. I meant like the ones that would accept the register at ports, um, once
Chris Chambers: Yeah.
Benjamin James Kuper-Smith: or twenty-five sorry.
Chris Chambers: at the moment.
Benjamin James Kuper-Smith: Yeah. So like one thing I was wondering about, like, it, doesn't it at some point. I was wondering like, why Jonas would want to do this because it seems to me at some point you have a, you know, you have a ranking of prestigiousness of the journals and then whoever's the most prestigious in a field is going to get basically all of the things I would imagine.
So like, if you're saying that human behavior says, okay, we're going to do this, you know, it has the nature label on it. So why would you then want to publish that in any other [00:46:00] journals? So it seems to be almost like. I don't know, as soon as there's like one journal, that's clearly more prestigious than the others that then suddenly the others.
Do you see what I mean? Like suddenly they kind of.
Chris Chambers: This transcends general rank. one of the, one of the, uh, It gives us, so let's, let's step back from it. What gives a journal? It's prestige has rank its mystique, generally prestigious journals claim implicitly or explicitly. That what makes us special that our review process is more rigorous. Our few processes better in some way, you know, and there's, there's this kind of underlying assumption that somehow the very same people doing the very same reason. We'll do it through a different standard based on the journal. And it's, I think it's a nonsense been particularly the case when these journals don't make the reviews available and there's no way of actually verifying that this is the case. Um, prestige is largely fake. Um, and one of the ways we can reveal it to be fake is by taking back control of the peer review process itself that the journal label is [00:47:00] simply the label.
Right? If you do, if you get a review through peer community and registered. All of the prestige and quality that is built into peer review. If you believe peer review does that is done by PCI and it's available, you can read it all. And then the journal is simply the output. If you want to stick it in the index scientific record, there it is.
And it might be changed for if you're a PCI, registered reports, friendly journal. Now you might think what's the incentive, as you said, why, why would a journal agree to this? Why would they syringe. Control of the peer review process like this to outside party. would they take this risk? And for this, you have to realize something about the culture of journals, which is that journals are run by editors and editors are primarily academics and academics are the same community forms PCI.
We're all the same people. Right. And this is something that I think is [00:48:00] really important to emphasize I'm talking to journal editors about peer community in register reports, I'm talking to the exact same people that built peer community in registered reports, we're all part of the same group. I've found that generally, this is, we're actually really excited about this they're not outsourcing. To somebody else they can join peer community and registered reports themselves as a recommender. I think of it like joining the European union, you're surrendering some sovereignty over your own law. In return for becoming part of a much larger community, which can have a much greater influence. you were to yourself in the position of a journal editor at a, at a high quality specialist journal in your area, I come to you and say, would you like to become a PCR, registered reports? Uh, in doing so, um, your journal will commit to accepting the recommendations of PCI are without further review in return you [00:49:00] can become a recommender with us and you can have a say over what gets published in other journals, right?
So we're taking all of that editorial expertise and we're pulling it a way that best for the community, and also works for the editors. I mean, the editors themselves. Gain influence over the scientific record that transcends the relatively borders of their little kind of.
know, their, their little journal that they've got.
And no journal is big, you know, very few journals and monsters, most journals are quite small and, and specialized and dedicated to their field and they can go beyond that now and have a much bigger influence. I found that, um, there's very little hesitation. Many journals and joining, I guess they don't see it as outsourcing.
They see it as insourcing. They see it as well. I can join and some of my editors can join. I gained expertise. Like I gained the benefit of a community of editors are already very good at editing registered reports and have a lot of experience. contribute my expertise. We can be part of something bigger [00:50:00] and we can also take back control up here. Which is something. Yeah. Editors, academic editors at journals. Don't love publishers. They do it because that's the only way. It's the only option available to them at the moment, but they have no love for publishers. So I think that's why we're seeing such support from PCI.
Benjamin James Kuper-Smith: By the way. Is there ever any consideration of PCI to have the engine with something like that or would that just be a silly, pointless thing or is it just a, not what you do or.
Chris Chambers: So the piece that PCI is aren't journalists themselves, there is a peer community journal, which was created recently, offers the simplest track for any, any researcher. So any article that's recommended by any PCI can be published. Without any changes whatsoever in the peer community journal for free, a diamond open access journal, um, and that's, and it's completely free for everyone and free [00:51:00] to access and read and everything. um, that's certainly an available option for researchers. Uh, but of course, is, this is the secret to making any change in the open. Kind of a world is you've got to, you've got to create options for people that are not too distant from what they're doing right now. got, you've got to create a path for people to use traditional journals in the way they're doing now, but in a better way. And this is where the other journals come in. So for example, can publish in cortex or Royal society, open science or BMJ, open science or any number of other journals that are on our register reports for the. In the same way as you can appear commute the peer community journal, but these are more recognized names.
And so obviously a, there's obviously a reason to create a transitional time where researchers might prefer to have their article published in a Royal society journal than in the peer community journal. you just create the options for people to choose. And then, and this is, you know, over time I suspect what will happen as PCI [00:52:00] registered reports continues to grow. And continues to become more popular is the name of the journal white medicine much because that's not where the stamp of quality comes from. You know, the peer review process is managed by P PCI, registered reports and the article in the journal links to those reviews, the name of the journal becomes kind of pointless.
It's just, it is there and it will benefit from. If that journal is more prominent than maybe more people will read it. beyond that, we were breaking the, this kind of fake prestige label somehow the name of the journal confers some magical quality on science, which I don't think it is.
Benjamin James Kuper-Smith: Yeah, no, I think I'm also fairly idealistic when it comes to these things. Every time I have like a pre-print of something, I think go. Do I need to submit this to journal. It's fine. The way, I mean, show, I want people to comment on it, you know, get some, some feedback from, from colleagues or whatever, but there's always part of me that goes like, maybe not [00:53:00] submit, maybe dancing with this one.
It's fine. Everyone can read it. It's a free print. That's out there.
Chris Chambers: that's because you're a scientist and this is what, uh, if we, I remember I'll never forget going to a conference back in 2015 in London, where we were talking about all these issues and, um, a certain physicist gave him very impressive talk, but he was like a traveler from the future.
Particle physics is 50 years ahead of. And they basically live in the world of preprints and happens at that stage. And all of the big scientific advances made at a pre-print stage and occasionally, you know, stuff that goes into journals, but doesn't really matter to them very much. And that's kind of where we want to be as well.
We don't need journals as signals of prestige quality. give it all it's available. We give, we are the source, the arbiters, the evaluators. We are everything. When it comes to quality, the journal is simply like a, some type setting and an index and a DOI.
Benjamin James Kuper-Smith: Yeah, I guess the, the, the idealistic [00:54:00] romantic ideal, uh, would probably be illegally penance Mon. Months, what is it solving that you might have authors, whatever it was, and then just posting on an archive and saying, okay, I'm done
Chris Chambers: Right.
Benjamin James Kuper-Smith: like that. I guess that is just like the optimal thing you can do. I have one kind of question.
Sounds negative, but it's not negative towards you, but more towards how slow process progresses, which is I was reading, you know, the editorials you wrote when various journals, you were part of introducing retro ports and that kind of stuff. And I said, I listened to your talks, that kind of stuff. And now you're talking to me and later you were talking in one home.
Do you ever get bored of talking about this?
Chris Chambers: Never, I never, I can talk about, I can talk your socks off to be honest. Um, I could talk about registered reports for eight hours without a break. No problem. It's one of those things that I just. I just think it's, um, it's got so much potential and I've only just started. We've only just started to unlock what it's capable of. It can go. I mean, the minute you take [00:55:00] out publication bias from site, You free, the shackles are dropped from so many different corners are so many different restraints and stupid and incentives in there, which don't help us. can, I think it opens so many doors and peer community registered reports is one of those, cause we've decided to tackle directly all of the negative points that have been raised.
All of the weaknesses of doing registered reports in the journal. We've tried to deal with with peer community and registry reports directly, and that's showing promising signs. so I, you know, I can talk about this all day and yes, there are frustrations. I'm always, I'm permanently, uh, impatient, you know, I'd want everything to happen.
Now I want more journals to be offering this format. Now I want, I want more early career researchers to have this option because I think it's liberating. There's so many researchers have. About the loss of anxiety they [00:56:00] experience when they use this format compared to the regular article process, and many authors at cortex have written to me and said, wow, I haven't done this.
I don't know that I'll ever be able to go back. It's kind of like if you had the option to fly business class, would you ever go back to. And that's how a lot of people feel. I think when they go through this process, they feel that it's, it's it's more constructive, it's not biased against certain findings.
And a lot of researchers stick with it and they decide that it's simply better for their career to do it. Um, and so I, then this is why I can talk about it endlessly, frankly, because I think we haven't even begun to scratch the surface of what it's capable of achieving.
Benjamin James Kuper-Smith: Yeah. I think the anxiety thing is that must as it. I haven't done it yet, but it must be a big thing because. I mean, of course you're partly nervous about the results when they come in, because you know, you want to be right. You know, you wanna, you want to have a correct theory, whatever the predict something, but there's, yeah, there's always this kind of thing like that, you know, if these results aren't interesting, then [00:57:00] you can say goodbye to certain journals, always going to be harder to, you know,
Chris Chambers: And that is frankly ridiculous that, that, that we, that we, we do this to ourselves, um, is, uh, it is just crazy. And I. It's something that people outside science don't understand, they don't think science works like this. They think that is right back to, where we talked about earlier. When I got into science as a, you know,
as a 20 year old, I didn't have ever imagined that.
That was the way science works. And needed it. Neither does anyone else, except people who are doctrinated and sort of become an institutionalized and they realize they have to play this silly game in order to keep going to keep doing science. And we've got to break it. We've got to smash it to bits.
And that's what, that's
Benjamin James Kuper-Smith: I
Chris Chambers: we're trying to do. That's what registered reports are all about.
Benjamin James Kuper-Smith: think that's a good note to end on then that's my stuff.