MSR - RR dates and timeline

[neilernst]

Tentative timeline, based on other MSR tracks:

• Pre-registration plans due: February 6, 2020
• Author notification: March 2, 2020
• 1 page summary plan / camera-ready: March 16, 2020
• Reports registered with OSF registry: Mar 31, 2020

The period between initial plan and decision is only 4 weeks, which is pretty short if the registration is complex.

[feigensp]

Since this is new, I think we should start earlier. Can we do it at the beginning of January, because we also include a "rebuttal" phase? And I also think that reviewers might be having more questions for us, so more time for them might also be necessary.

[neilernst]

How about

• Jan 10 plans due
• Jan 31 initial reviews
• Feb 14 rebuttals/clarifications due
• Mar 2 final decision notifications
• (rest as above)

[feigensp]

Yes, that sounds good.

[snadi]

I think the revised timeline sounds good to give you more time in the middle. Did any of you already try out OSF?

[neilernst]

We have been looking into the format. I think for now we will have a custom registration template, as the OSF one is pretty weighty. Then if accepted we ask the authors to put the registration on OSF.

[snadi]

Sounds good. Is there a backup plan if your custom template doesn't go through? Or perhaps I misunderstood: do you need some form of OSF approval to create a custom template?

[neilernst]

hmm, good question. @feigensp is working on the CFP/template. I think our model was the authors would then upload the registration using the "IPA" approach in the screenshot (but I will check with OSF).

[snadi]

So from that snapshot, it seems this is supported by default by the platform and doesn't need any approval from their part. I guess you just need to figure out the questions you want the authors to answer. So I guess it's all OK :-)

[neilernst]

Ok, in that case I’ll close this issue and you can use these dates for the MSR website - let me know if you want a different format or via email.

Date	Milestone
January 10, 2020	study protocols and plans due
January 31, 2020	initial protocol reviews
February 14,2020	rebuttals/clarifications due
March 2, 2020	In Principle Acceptance (IPA) decision notifications
March 16, 2020	1 page summary plan / camera-ready
March 31, 2020	Reports registered with OSF registry

[snadi]

Just a clarification... why is it in principle acceptance? Aren't you simply accepting the registered report, not actually the study yet? Or is this the typical term used for registered reports?

[neilernst]

That’s the typical term used. In principle it can be accepted if the protocol is followed.

…

On Sep 19, 2019, at 15:18, snadi ***@***.***> wrote: Just a clarification... why is it in principle acceptance? Aren't you simply accepting the registered report, not actually the study yet? Or is this the typical term used for registered reports? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[robertfeldt]

One question (I think from Laurie Williams) when I mentioned pre-registration/RRs in my ESEM19 keynote:

"How are changes to be handled that come from comments on the pre-reg report when it is presented at MSR?"

I guess comments come either from reviewers during Stage 1 review but can also come from audience when presented at MSR. The former are handled in rebuttal/revision process pre MSR accept? While changes that happens between that and Stage 2 review must be explicitly explained and clarified in the paper submitted for Stage 2 review?

[neilernst]

That’s an excellent question. One solution would be open review but I believe @snadi looked at this and it didn’t meet the needs. The problem is how we check that a study actually responded to the MSR comments (and which comments, sorry, ‘questions’ are worth responding to). If the protocol is deviated from after IAP I think it can still be published, but would need more fulsome review.

…

On Sep 20, 2019, at 11:45, Robert Feldt ***@***.***> wrote: One question (I think from Laurie Williams) when I mentioned pre-registration/RRs in my ESEM19 keynote: "How are changes to be handled that come from comments on the pre-reg report when it is presented at MSR?" I guess comments come either from reviewers during Stage 1 review but can also come from audience when presented at MSR. The former are handled in rebuttal/revision process pre MSR accept? While changes that happens between that and Stage 2 review must be explicitly explained and clarified in the paper submitted for Stage 2 review? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ctreude]

Would it be feasible to use (some of) the same reviewers for the versions submitted to MSR and the versions submitted to EMSE? And making comments made during the presentation at MSR available to the EMSE reviewers?