-
|
From the crowdlab paper: "Illustrating how many real-world multiannotator datasets look, Figure 1 shows a disparity in annotator quality as well as many examples whose consensus |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Hi @vahuja4, thanks for the question! Even if you had 3 annotators (or any constant number) annotating all the data, CROWDLAB will still be helpful in providing more robust consensus labels for your dataset. This is because in addition to the annotations from your annotators, CROWDLAB utilizes information (predictions) from a trained classifier to determine the best consensus label for each example. The CROWDLAB algorithm will automatically weigh the trustworthiness of the annotators and classifier and ensemble their annotations/predictions to provide you a consensus label, alongside a score for each example indicating the confidence level of that consensus label. This process is similar regardless if you have a constant or varying number of annotators for each example. In fact, Appendix C of the CROWDLAB paper showcases CROWDLAB in use with a dataset where each example was annotated by 50 annotators! Lastly, you can use any trained classifier with CROWDLAB, but as with most of the cleanlab package, a better classifier would likely yield better results 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
@huiwengoh - thank you for your reply! I have one more question: for the crowdlab algorithm to work, we need the following:
Can you please confirm? |
Beta Was this translation helpful? Give feedback.
-
|
You are mostly on the right track! But one key details is that CROWDLAB does not require ground truth labels, in fact in most applications the ground truth labels will not be available to you. Rather, you can obtain out-of-sample predictions by your classifier for all your datapoints by performing cross validation, and then CROWDLAB can provide you with consensus label and scores for all those datapoints as well. Here is a quick tutorial that might make that process clearer to you, hope it helps and please let us know if you have any other questions! |
Beta Was this translation helpful? Give feedback.
Hi @vahuja4, thanks for the question!
Even if you had 3 annotators (or any constant number) annotating all the data, CROWDLAB will still be helpful in providing more robust consensus labels for your dataset.
This is because in addition to the annotations from your annotators, CROWDLAB utilizes information (predictions) from a trained classifier to determine the best consensus label for each example. The CROWDLAB algorithm will automatically weigh the trustworthiness of the annotators and classifier and ensemble their annotations/predictions to provide you a consensus label, alongside a score for each example indicating the confidence level of that consensus label.
This process is similar …