Skip to content

Instance re-distribution after incomplete submissions #100

@jl3676

Description

@jl3676

Hi, I recently used potato to collect annotation data on Prolific and I set the labels_per_instance variable to a fixed number. I ran into the issue that incomplete submissions (some workers didn't finish all the instances assigned to them) remained in the instance distribution counts, so after data collection stopped (new workers were told that they came too late and there were no tasks remaining, so they must return their submissions), I didn't get as much data as I wanted per instance sometimes. I wonder if there is a known solution to this problem? Additionally, for the participants who "came too late," being told that could be frustrating because it could feel like a waste of their time (according to some of the messages they sent me).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions