Tutorial: Using Amazon SNS with Amazon MTurk

Amazon Mechanical Turk (MTurk) is a marketplace for human tasks, such as generation of training data for machine learning. Customers who complete these tasks are called Workers and customers who publish these tasks are called Requesters.

Requesters can use the MTurk Requester Application Programming Interface (API) to automate the submission of Human Intelligence Tasks (HITs) and collection of their results. This tutorial will cover how to use Amazon Simple Notification Service (SNS) with the API to set and get notifications from MTurk about the state of your tasks. SNS makes it easier to automate steps you in workflow. For example, you can write code that automatically retrieves results from MTurk the moment a Worker completes an assignment for one of your HITs. SNS is one way to receive Notifications from MTurk. Amazon Simple Queue Service (SQS) is another that is covered in this tutorial.

Getting Started
To follow this tutorial, we will assume that you have some familiarity with the Requester API and Python. If you are unfamiliar with the API, check out this primer which will also help you set up and configure the accounts you need to make API calls.

Create an SNS Topic
First, let us start by setting up an SNS topic. Log in to your AWS account from https://aws.amazon.com and then navigate to the SNS console:

Click on “Create topic” and pick a name for it:

Great! Now you have a new SNS topic set up. You should see it show up in your console like below. Copy the “ARN” of your topic and keep it handy, as you will need it shortly.

Next, MTurk needs permission to be able to push notifications to your new topic. From the SNS console, select your topic and click on the “Actions” dropdown menu and select “Edit topic policy”:

Switch to the “Advanced” tab:

Swap in your SNS topic ARN into the template policy below. You can also optionally add the account ID of the AWS account you use to make calls to MTurk in the “aws:SourceAccount” field. This is a security precaution to ensure that MTurk will send notifications to your SNS topic for HITs created from your account only.

{
"Version": "2012-10-17",
"Id": "YOUR_SNS_ARN/MTurkOnlyPolicy",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "mturk-requester.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "YOUR_SNS_ARN",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "YOUR_AWS_ID"
},
"Bool": {
"aws:SecureTransport":"true"
}
}
}
]
}

After swapping in the details in bold, paste the policy code into the Advanced view text box and click on “Update policy”. The SNS ARN in today’s tutorial is arn:aws:sns:us-east-1:217388664551:track-hits. Your AWS ID can be retrieve at this page and is a series of numbers like 123456789012.

In order to see or use the notifications coming from an SNS topic, you need to create a subscription to it. The simplest subscription you can create is an email notification. When your SNS topic gets something new from MTurk, it will forward it to your email. Let us set that up now for testing.

From the SNS console, select the topic again and from the Actions menu select “Subscribe to topic” :

Change the Protocol to “Email” and enter in your email address:

Amazon will send you a confirmation email and once you click on it to confirm, your email will receive all notifications from your SNS topic.

Now you have an SNS topic that is set up and ready to receive notifications from MTurk! Each time we receive a notification, SNS will trigger an e-mail to the address you supplied above (myemail@myemail.com in this case). Next, let’s create a task on MTurk and configure it to send notifications. We will start with the exact same sample code used with in our basic primer on using Python with MTurk (note that you can use wide range of languages and AWS SDKs to access MTurk via the Requester API).

# Read in question content from questions.xml
question = open(name='questions.xml',mode='r').read()
new_hit = mturk.create_hit(
Title = 'Is this Tweet happy, angry, excited, scared, annoyed or upset?',
Description = 'Read this tweet and type out one word to describe the emotion of the person posting it: happy, angry, scared, annoyed or upset',
Keywords = 'text, quick, labeling',
Reward = '0.15',
MaxAssignments = 1,
LifetimeInSeconds = 172800,
AssignmentDurationInSeconds = 600,
AutoApprovalDelayInSeconds = 14400,
Question = question,
)
print "A new HIT has been created. You can preview it here:"
print "https://workersandbox.mturk.com/mturk/preview?groupId=" + new_hit['HIT']['HITGroupId']
print "HITID = " + new_hit['HIT']['HITId'] + " (Use to Get Results)"
# Remember to modify the URL above when you're publishing
# HITs to the live marketplace.
# Use: https://worker.mturk.com/mturk/preview?groupId=

Now add the following lines to bottom of the sample after swapping in your SNS topic:

mturk.update_notification_settings( 
HITTypeId = new_hit['HIT']['HITTypeId']
Notification = {
'Destination': 'YOUR_SNS_TOPIC',
'Transport':'SNS',
'Version':'2014-08-15',
'EventTypes':['HITReviewable']
}
)

As you can see, we are calling the “update_notification_settings” operation — this operation uses the “HITTypeID” for the HIT type associated with the task you just created. HIT types are templates for HITs in MTurk. They can be used to create new HITs without having to specify the price, title and so on each time. You can create them directly or MTurk will generate them for you when you create a new HIT using the CreateHIT operation like in the sample above.

Note the “EventTypes” field. This is an array where you can specify all the different types of events that you want MTurk to notify your SNS topic about. We are specifying “HitReviewable” but there are up to 12 types of events that you can follow at once -

AssignmentAccepted: a Worker accepts an Assignment
AssignmentAbandoned: a Worker accepts an Assignment but does not complete it in time
AssignmentReturned: a Worker presses the “Return” button and returns the Assignment after first accepting it
AssignmentSubmitted: a Worker completes and submits an Assignment
AssignmentRejected: an Assignment has been rejected. Assignments can be rejected by you, but they can also be automatically rejected if you are using an automated MTurk Review Policy.
AssignmentApproved: an Assignment has been approved. Assignments can be approved by you, but they can be automatically approved after a specified time or if you are using an automated MTurk Review Policy.
HITCreated: a HIT has been created. 
HITExtended: the expiration date for a HIT has been extended. This is usually done by you, but can also be happen if you are using an automated MTurk Review Policy.
HITDisposed: a HIT has been deleted
HITReviewable: all of the Assignments for a HIT have been submitted. For example if you asked three Workers to label an image, the HIT is reviewable once all three Workers have submitted their labels.
HITExpired: a HIT has expired
Ping: this is used for testing with the send_test_event_notification call

Now if you run your code sample you should have a new HIT created that comes with a notification configured for “HITReviewable”.

If you now log in to the Worker Sandbox with a test Worker account and complete your HIT, you should receive a notification in your email inbox through SNS that your HIT is ready for review:

As you can see, the notification message comes with a variety of useful fields, such as the HITId and the type of event that triggered the notification.

That’s it! Once you have this working, you can start exploring more powerful ways to use SNS beyond email notifications. You can configure SNS to send you notifications to an HTTP endpoint that you create. You can also make SNS trigger an AWS Lambda function that can parse the notification and use the HITId to call MTurk. Since MTurk is supported natively in AWS Lambda this can be a very quick way to build automated workflows to submit and retrieve results.

Lastly, MTurk also supports notifications to AWS Simple Queue Service (SQS). As we shared earlier, this tutorial covers how to set that up.

We hope you find this tutorial useful. If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.

Responses
The author has chosen not to show responses on this story. You can still respond by clicking the response bubble.