Tutorial: Best practices for managing Workers in follow-up surveys or longitudinal studies
If you’ve ever posted a survey or study on Amazon Mechanical Turk, you’ve probably needed to re-post a survey. Either you changed the audience, found a mistake you made, or want to collect more responses from new Workers. Posting the survey again with MTurk is easy, but what if you want to include or exclude the Workers that responded to your survey the first time around? In this tutorial we’ll explain how to restrict Workers from participating in your survey when you post it a second time. We’ll also look at how you can re-survey Workers that responded to your survey the first time.
To exclude Workers, you will want to use a Qualification. Qualifications are commonly used to identify Workers that have have been “qualified” as being good at a task. Here, we will use Qualifications to create a list of Workers that we’ll exclude.
A word of caution: MTurk has a built-in capability for a Requester to “block” a Worker from working on that Requester’s tasks in the future. Requesters use this capability to identify and exclude Workers that have consistently produced poor quality results on their tasks. We strongly urge Requesters to avoid using this feature to exclude Workers for convenience. Doing so could negatively impact the Worker’s reputation in MTurk.
To start, make sure you have a list of the Workers who completed your survey the first time. Go to the Manage tab and select Qualification Types, then hit Create New Qualification Type.
We’ll call this Qualification “Completed My Survey Already”. You can choose any name you like, but be aware that Workers can see Qualifications they have been granted and the Qualification name will display as a requirement to work on your task. Type a description and hit “Create”.
Your Qualification will show up in the list of Qualifications. When it does, go to the Workers tab and select Download CSV on the right.
When your file is ready, you can then download a CSV file of your Workers that you can open in Excel or Google Sheets.
This file contains all of the statistics and Qualifications for all of the Workers that have worked for you in the past. You can update this file with new values for any Qualifications and upload it to MTurk to set new values. To make this easy, we’ll delete any Workers we’re not interested in and all of the columns except for “UPDATE-Completed My Survey Already” which corresponds to the new Qualification. Any values provided here will become the new Qualification values for the Workers we include. Here we will give a value of “1” to the five Workers we want to exclude from our next survey.
Go back to the Worker tab and select the Upload CSV link to upload the new file you’ve created.
After the file completes uploading you can confirm the Qualification changes.
When we return to the Qualification tab we now see that there are five Workers assigned to the Qualification “Completed My Survey Already”.
Now we just need to exclude these Workers from our next batch. Select Edit for your project.
Go to the Worker Requirements section of your Project Properties and add the new Criterion you’ve created. You’ll note that we’ve specified we only want to request responses from Workers that have NOT been granted the Qualification “Completed My Survey Already”.
Hit Save and you’re all set to republish your survey.
If you are doing a longitudinal study and wanted to include only Workers that participated in your survey in the past, instead of excluding them, you would change the requirement above to “has been granted”.
If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.
