Launch: Requesters now have more control over HIT visibility via the MTurk API

Amazon Mechanical Turk (MTurk) is made possible by a large and diverse community of Worker customers around the world. These customers contribute time and talent every day to help make Requester customers successful.

One mechanism that Requesters use to find the best suited Workers to complete their tasks are Qualifications. Qualifications work by letting Requesters specify criteria that Workers must meet to be eligible to complete that Requesters task. For example, a Requester might use the built-in MTurk Locale Qualification to specify that they need Workers in the United States or Canada, and a System Qualification to indicate they prefer Workers who have completed and received approval on at least 50 MTurk Human Intelligence Tasks (HITs) already. You can learn more about MTurk Qualifications from this tutorial published last year.

Introducing more flexibility for QualificationRequirements

Today, we’re pleased to introduce additional controls for Requesters using the MTurk Application Programming Interface (API). Now, Requesters can not only specify the requirements that Workers must meet to be eligible to complete the Requester’s tasks, but the visibility of these tasks.

Using the ActionsGuarded field on the QualificationRequirement data structure. This field allows for control over how Workers who don’t yet possess the required Qualifications are able to interact with the HIT. For example, A Requester can prevent Workers without the required Qualifications from previewing the HIT by setting ActionsGuarded to ‘PreviewAndAccept’ or prevent Workers from seeing the HIT altogether by setting ActionsGuarded to ‘DiscoverPreviewAndAccept’.

This new capability gives Requesters more control over which Workers can discover, preview, and accept their HITs. It also improves the Worker experience, by reducing cases of Workers discovering HITs that they won’t or can’t qualify for. For instance, a Requester who creates HITs only for Workers in India might set ActionsGuarded to ‘DiscoverPreviewAndAccept’ to avoid Workers in other countries (who will be ineligible for the HIT) to discover it in their search, filter, or browse results. This removes unnecessary results from Workers outside of India (who don’t meet the Qualification), while keeping the experience for Workers in India, who do meet the Qualification, unchanged.

Demonstrating this with code samples

To illustrate these new capabilities, we will use the Ruby programming language. The concepts shown below, however, work in any of the languages support by the AWS Software Development Kit and Command Line Interface. You can learn more about these here.

We’ll begin by installing the latest ruby gem for AWS:

gem install aws-sdk

Once installed, we’ll create a file (in our case, called create_hit.rb) and add the following code to it (the code is taken from the Developer Getting Started Guide on the MTurk Requester Website):

require 'aws-sdk'

aws_access_key_id = 'YOUR_ACCESS_ID'
aws_secret_access_key = 'YOUR_SECRET_KEY'

endpoint = 'https://mturk-requester-sandbox.us-east-1.amazonaws.com'
region = 'us-east-1'

# Uncomment this line to use in production
# endpoint = 'https://mturk-requester.us-east-1.amazonaws.com'

credentials = Aws::Credentials.new(aws_access_key_id, aws_secret_access_key)
mturk = Aws::MTurk::Client.new(endpoint: endpoint, credentials: credentials, region: region)

You will want to replace YOUR_ACCESS_KEY and YOUR_SECRET_KEY above with your AWS Access Key and AWS Secret Key. If you don’t know where to find those, check out this blog post.

Next, we’ll add an example question to the bottom of that file. Our question will simply ask the Worker about their day, but request that they respond in Spanish:

hit_question = <<EOF
<HTMLQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2011-11-11/HTMLQuestion.xsd">
<HTMLContent><![CDATA[<!DOCTYPE html>
<html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'/>
<script type='text/javascript' src='https://s3.amazonaws.com/mturk-public/externalHIT_v1.js'></script>
</head>
<body>
<form name='mturk_form' method='post' id='mturk_form' action='https://www.mturk.com/mturk/externalSubmit'>
<input type='hidden' value='' name='assignmentId' id='assignmentId'/>
<h1>Are you having a great day so far?</h1>
<h3>¿Cuál es la respuesta?</h3>
<p><textarea name='answer' cols='80' rows='3'></textarea></p>
<p><input type='submit' id='submitButton' value='Submit' /></p>
</form>
<script language='Javascript'>turkSetAssignmentID();</script>
</body>
</html>]]>
</HTMLContent>
<FrameHeight>450</FrameHeight>
</HTMLQuestion>
EOF

We will then add the following code to create a HIT. Just add this code to the bottom of your file:

result = mturk.create_hit(
lifetime_in_seconds: 60 * 60 * 4,
assignment_duration_in_seconds: 120,
max_assignments: 1,
reward: '0.25',
title: 'My Spanish HIT (for any Worker)',
description: 'Answer an English question with a Spanish answer',
question: hit_question,
)
print "https://workersandbox.mturk.com/projects/#{result.hit.hit_type_id}/tasks\n"

To run this code, simply type the following at the terminal command line:

ruby create_hit.rb

You should see the following output:

https://workersandbox.mturk.com/projects/3A98TDU8OEOJUIABU2CDBALOJTNF3E/tasks

If you click that link, it should take you to view your HIT. It should look similar to this:

You’ve successfully created a HIT without any Qualification Requirements. Next, we’ll begin adding Qualification Requirements with ActionsGuarded to see how they perform.

Controlling the ability to Accept a HIT

First, let’s change or add the lines shown in bold below in your code:

LOCALE_QUALIFICATION_ID = "00000000000000000071"
NUM_HITS_APPROVED_QUALIFICATION_ID = "00000000000000000040"
qualification_requirements = [
Aws::MTurk::Types::QualificationRequirement.new(
qualification_type_id: LOCALE_QUALIFICATION_ID,
comparator: 'EqualTo',
locale_values: [ { country: 'ES' }],
actions_guarded: "Accept"
)
]
result = mturk.create_hit(
lifetime_in_seconds: 60 * 60 * 4,
assignment_duration_in_seconds: 120,
max_assignments: 1,
reward: '0.25',
title: 'My Spanish HIT (for ES Workers)',
description: 'Answer an English question with a Spanish answer',
question: hit_question,
qualification_requirements: qualification_requirements
)

The qualification_requirement variable above defines the criteria that a Worker must meet to work on your HIT. Above, we’ve used the MTurk built-in Locale Qualification to select only Workers who are located in Spain. We’ve added the qualification_requirements variable to the call to CreateHIT. We’ve changed the title to identify this HIT separately from the previous HIT.

Now, let’s run our Ruby program again:

ruby create_hit.rb
https://workersandbox.mturk.com/projects/39E5EJIUG2RREAJ9SSHKT9QRTFCL9AY/tasks

When we open the link above, we should see the following:

Although the title has changed to say (for ES Workers) nothing else seems different. But if we click Accept and try to work on it, unless we are in Spain we should see this message:

We’ve successfully created a HIT with a Qualification. Nice work!

Controlling the ability to Preview and Accept a HIT

But what if we didn’t want Workers to be able to preview the HIT? For that, let’s change the lines below in bold in our code:

qualification_requirements = [
Aws::MTurk::Types::QualificationRequirement.new(
qualification_type_id: LOCALE_QUALIFICATION_ID,
comparator: 'EqualTo',
locale_values: [ { country: 'ES' }],
actions_guarded: "PreviewAndAccept"
)
]
result = mturk.create_hit(
lifetime_in_seconds: 60 * 60 * 4,
assignment_duration_in_seconds: 120,
max_assignments: 1,
reward: '0.25',
title: 'My Private Spanish HIT (for ES Workers)',
description: 'Answer an English question with a Spanish answer',
question: hit_question,
qualification_requirements: qualification_requirements
)

Now, let’s create the HIT again and visit the URL. We should see a message similar to the following:

And if we were to search for the HITs, by typing a portion of our title into the search bar (e.g., ‘My Private Spanish HIT’) we should see the following:

Notice that we are able to search and find the HIT, and we can see the HIT properties such as Title, Description, and more. But the Preview button is disabled and we can’t access the task itself.

Controlling the ability to Discover a HIT

What if we don’t want Workers to discover the HIT at all unless they have the Qualifications required? For that, we’ll change the lines of code shown below in bold:

qualification_requirements = [
Aws::MTurk::Types::QualificationRequirement.new(
qualification_type_id: LOCALE_QUALIFICATION_ID,
comparator: 'EqualTo',
locale_values: [ { country: 'ES' }],
actions_guarded: "DiscoverPreviewAndAccept"
)
]
result = mturk.create_hit(
lifetime_in_seconds: 60 * 60 * 4,
assignment_duration_in_seconds: 120,
max_assignments: 1,
reward: '0.25',
title: 'My Hidden Spanish HIT (for ES Workers)',
description: 'Answer an English question with a Spanish answer',
question: hit_question,
qualification_requirements: qualification_requirements
)

When we run our script, and we access the link, we should see the same message that we saw earlier when we tried to preview it:

But we’ll also find that the HIT won’t appear in search results at all. Try searching for it using the title ‘My Hidden Spanish HIT’:

Unless you are located in Spain, you should see a screen similar to the one above.

Wrapping up

That’s it. Congratulations! You’ve successfully created a HIT using a Qualification Requirement that restricts the ability to Discover, Preview, and Accept unless the Worker meets the Qualification Requirement.

If you’d like to learn more about ActionsGuarded and QualificationRequirements in general, check out the MTurk documentation: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_QualificationRequirementDataStructureArticle.html

We hope you found this useful. If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.

Responses
The author has chosen not to show responses on this story. You can still respond by clicking the response bubble.