Tutorial: Understanding Requirements and Qualifications

Amazon Mechanical Turk (MTurk) is made possible by a large and diverse community of Worker customers around the world. These customers contribute time and talent every day to help make Requester customers successful .When using MTurk one of the early questions Requesters ask is “how do I specify who does my work?” This tutorial will help explain the tools at your disposal for defining the audience of Workers that can work on your HITs.

For this tutorial we’ll focus on how to use the MTurk Requester website to create HITs with different Qualification Requirements. All of the concepts also apply to HITs created from the API but you will want to reference the API documentation here and here for more detail. The API documentation also provides information on creating Qualification Tests. These are a method to prompt Workers to successfully answer test questions before they can Accept your HIT.

Worker Requirements

When you create a new Project from the MTurk Requester website the first page prompts you to provide requirements for the Workers that do your task.

The first option is whether or not you want to only use Masters Workers. Masters Workers have consistently demonstrated a high degree of success in performing a wide range of HITs across a large number of Requesters. For most tasks you don’t need to use Masters but it’s an option you have available. Note that there is an additional 5% fee when using Masters Workers.

The second option is to add additional criteria to your task through Qualifications. When you select Add another criterion a drop down is added where you can select qualifications for your task.

The first options are System Qualifications for Location, Approval Rate, and Number of HITs approved. After that are the Premium Qualifications which are qualifications specifying demographic and behavioral profiles of different Worker groups. Finally, there are Qualifications you’ve created yourself. We’ll discuss each type below.

The final option is the Adult Content Qualification. When checked, your task is only available to Workers who are willing to see potentially offensive content. Many Requesters that use MTurk for image moderation or other similar tasks use this option. More detail on this can be found here.

System Qualifications

There are three system Qualifications you can use. These Qualfications are based on Worker Registration and activity information.

Location

You can restrict Workers based on the country or US state that they registered from. For example, if you wanted to request inputs from Workers in the United Kingdom you would start by selecting the Location Qualification, setting the Comparator to is, and selecting United Kingdom in the dropdown:

If you wanted to select multiple countries and include both the United Kingdom and United States, you would change the Comparator to is one of and shift- or ctrl-click both the United Kingdom and United States in the list.

You can also exclude locations by selecting is not or is not one of to exclude selected items. For example, the following would exclude Workers from the United States.

It’s also possible to stack Qualifications by adding additional criteria. If we wanted to include Workers from every US state but exclude Workers in Minnesota, we would have an is Location Qualification for the United States and an is not Location Qualification for Minnesota.

When you have multiple Qualifications they are treated as AND operations so both must be met for a Worker to accept this task. Be careful when applying multiple Qualifications to ensure that it’s possible for a Worker to meet the requirements. For example, if we set the requirements below, no Worker would be eligible since it’s impossible to have registered in both the United States and Canada. The correct way to include Workers from both countries is to use the is one of Comparator as described above.

HIT Approval Rate

The Approval Rate is the rate that Requesters have approved HITs that Workers complete. For example, if a Worker has completed 500 HITs and had their work rejected 5 times, their approval rate is 99%. Note that each Requester has their own thresholds for approving or rejecting work so be careful not to set your rate too high or you may unnecessarily exclude Workers who would be good at your task.

Number of HITs Approved

This is the number of HITs that a Worker has successfully completed since they registered for MTurk. Many Requesters use this as a way of denoting that they want Workers with a certain level of experience as MTurk Workers.

Premium Qualifications

Many Requesters use MTurk as an audience for surveys. To help narrow your audience to Workers who fit a particular demographic or behavioral profile we have added Premium Qualifications that reflect self-reported information about the Workers on MTurk. There are dozens of Qualifications ranging from age and gender, to exercise frequency and the type of smartphone Workers own. These Qualifications are simpler than the System Qualifications and simply allow you to select if you want to include or exclude Workers that meet a certain profile. For example, the following would restrict the survey audience to Workers who own an iPhone and aren’t married.

Note that there are additional fees for using Premium Qualifications. More information can be found at https://requester.mturk.com/pricing.

Qualifications You Create

The final group of Qualifications are those you can create yourself. These are a great way to identify Workers you want to include or exclude from future work.

For your Qualifications you assign Workers a score, typically 0 to 100. How you score Workers is entirely up to you. Many Requesters simply assign a 1 or 0 depending on whether or not they want to include or exclude Workers. Others assign a value based on a Worker’s accuracy on previous HITs. How you score your Workers depends on how you are structuring your project.

When you add your Qualification to your task, there are a wide range of comparators you can use to filter.

Options include:

  1. To only include those Workers who have the Qualification (any value) use the has been granted comparator.
  2. It include all Workers EXCEPT those that have been granted the qualification use the has not been granted comparator.
  3. To restrict Workers based on the value they’ve been assigned, use the equal to, not equal to, greater than, greater than or equal to, less than, or less than or equal to comparators. You can also select or exclude multiple values by using the is one of or is not one of comparators.

Creating a new Qualification

To create a new Qualification go to the Manage tab and select Qualification Types.

Now select the Create New Qualification Type button.

We’ll call this Qualification “Good At Animal Identification” but you can pick any name you like. Just be aware that Workers can see the Qualifications they’ve been granted and the name will show up as a requirement to work on your task.

It will take a few minutes for your Qualification to show up in the list of Qualifications. When it does, go to the Workers tab and select Download CSV on the right.

When your file is ready, you can then download a file of all of your Workers that you can open in Excel or Google Sheets.

This file contains all of the statistics and Qualifications for all of the Workers that have worked for you in the past. You can update this file with new values for any Qualifications and upload it to MTurk to set new values. To make this easy, we’ll delete any Workers we’re not interested in and all of the columns except for “UPDATE-Good At Animal Identification” which corresponds to the new Qualification we created. Any values we provide here will become the new Qualification values for the Workers we include. Here we will give a value of “1” to the Workers that did well in our evaluation task.

Go back to the Worker tab and select the Upload CSV link to upload the new file you’ve created.

After the file completes uploading you can confirm the Qualification changes.

After the Qualification has been created you will be able to add it as criteria:

HIT Visibility

If you’ve specified any criteria for your HIT you have the option of deciding it’s visibilty. For many Requesters is easiest just to leave the task marked Public but there are other options.

If your HIT is marked Public, any Worker can preview your task, even if they aren’t qualified. If you mark the task as Private, all Workers will see your task in the list of available work but only those Workers that have the required Qualification will be able to Preview the contents of the task. This is useful if you want to limit the number of Workers that can see the information in your tasks. Hidden is used to hide HITs from showing up in the list of available work unless the Worker has the required Qualifications.

Qualifications in practice

There are a variety of ways that Requesters have discovered to choose Workers using Qualifications. We’ve described a some methods in the Tutorials below but there are a variety of other cases that we’ll capture in future Tutorials.

If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.

Responses
The author has chosen not to show responses on this story. You can still respond by clicking the response bubble.