Replies: 2 comments 2 replies
-
|
Hi @Rianne-vD , the SAFE procedure has some steps to ensure 'safety' in the process, using one person to screen and ASReview to speed up the process. If you add more people to the process, there's the benefit of an extra set of eyes to verify these steps. But also the complexity that every additional person will potentially see different items and make different decisions. Using the datatools extension you can quickly get an overview of these differences. I hope this helps. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Rianne-vD. Essentially, yes. I recommend checking IRR and using the example report is a nice tool. If/when raters do not see the same articles, we can see this as a missing data problem and the tool has additional options for calculating IRR with missing data. |
Beta Was this translation helpful? Give feedback.
-
|
Dear @bethgr, thank you for your reply. Would you recommend doing this in either the first or second step of the SAFE procedure, or both? |
Beta Was this translation helpful? Give feedback.
-
|
Both when possible. |
Beta Was this translation helpful? Give feedback.
-
We use ASReview in several projects, and I wonder if and how to measure interrater reliability (IRR) when scanning the titles and abstracts. Although the SAFE procedure is clearly explained, it does not mention anything about agreement between raters.
Given that ASReview presents new articles based on previous decisions, I assume that not all raters will see the same articles. This raises the question of how to effectively measure agreement in this context. I came across another discussion on GitHub about this topic and an example report. Do you recommend the implementation of checking the IRR / agreement rate over a new batch of articles as an additional step between the training phase and the active learning phase as described in the SAFE procedure?
Beta Was this translation helpful? Give feedback.
All reactions