-
Notifications
You must be signed in to change notification settings - Fork 65
Description
Hi, I recently used potato to collect annotation data on Prolific and I set the labels_per_instance variable to a fixed number. I ran into the issue that incomplete submissions (some workers didn't finish all the instances assigned to them) remained in the instance distribution counts, so after data collection stopped (new workers were told that they came too late and there were no tasks remaining, so they must return their submissions), I didn't get as much data as I wanted per instance sometimes. I wonder if there is a known solution to this problem? Additionally, for the participants who "came too late," being told that could be frustrating because it could feel like a waste of their time (according to some of the messages they sent me).