Qualifications and Worker Task Quality

Amazon Mechanical Turk (MTurk) Requesters consistently tell us that using Qualifications is one of the most effective strategies for optimizing the quality of results on Amazon MTurk. So, if you’re an MTurk Requester and unfamiliar with how to implement Qualifications, this post will help you understand why most Requesters use Qualifications, and direct you to detailed tutorials that will help you leverage Qualifications for your MTurk projects. While Qualifications require some time investment during task creation, Requesters tell us the investment is worthwhile as their projects produce better results.

Using Qualifications allows you to select or create specific Worker eligibility requirements for your projects. This enables you to direct projects to specific groups of Workers on MTurk. When you select a Qualification, it means that you can include or exclude Workers who meet your Qualification standard and control which workers can contribute to your project.

MTurk offers three types of pre-defined Qualifications — Masters, System and Premium Qualifications.

Masters are a group of Workers with demonstrated accuracy across multiple types of tasks on MTurk. Including the Masters Qualification as a requirement is one way to ensure that your project is completed by Workers who have consistently produced quality results.

System Qualifications can be leveraged to direct your tasks to experienced Workers. To direct your tasks to more tenured Workers, use the “Number of HITs Approved” System Qualification. Based on Requester feedback, depending on the type of task, you may want to set the “Number of HITs Approved” requirement at or above 10,000 approved HITs. This means that only Workers who have completed 10,000 approved HITs will have access to your project.

Equally important to targeting the most experienced Workers, setting your “HIT Approval Rate” System Qualification to greater than 98% will direct your projects to Workers who have consistently produced high quality tasks. The Hit Approval Rate represents the proportion of completed tasks that are approved by Requesters. Depending on the type of task (surveys for example), you may want to set the HIT Approval Rate above 98% to enforce even higher quality.

Requesters have seen that Number of HITs Approved (above 10,000) and HIT Approval Rate (above 98%) have a significant effect on the overall quality of results. You can also use System qualifications to select Location. If you are targeting Workers in the United States for example, you would use the Location Qualification drop down.

Premium Qualifications are typically used to select more granular Worker attributes such as income or job function. Keep in mind that selecting multiple premium qualifications can reduce your pool of Workers.

How can you implement these Qualifications?

MTurk Requester Site: If you are creating a HIT directly on the MTurk Requester site, you can select qualifications under the Worker Requirements section when you create a task or project. Read more about selecting pre-defined Qualifications or creating your own custom Qualifications.

MTurk Requester API: If you are using a Requester API, you can learn more about QualificationRequirement and access Qualification TypeIDs in MTurk’s API reference and this tutorial on how to select and create Qualifications. Please note that we will deprecate legacy Requester APIs and final support will end on June 1, 2019.

MTurk Qualifications are designed to help you target the Workers you want for your project, and when used correctly, enable you to select Workers with the attributes you need to deliver consistently reliable task quality for your projects.