Tutorial: Getting great survey results from MTurk and SurveyGizmo

Amazon Mechanical Turk
Happenings at MTurk
7 min readAug 21, 2017

--

MTurk is a great way to get fast responses to your survey or study but it works a little differently from traditional survey services. Today, we’re going to talk about how to manage the results of surveys you host on SurveyGizmo. You can also host your survey on MTurk itself but some customers choose to use SurveyGizmo, Qualtrics, or a number of other survey tools they may be more familiar with. To learn how to use MTurk with other survey services, check out these past posts about Qualtrics, Survey Monkey, and Google Forms.

Getting Started

To post a survey on MTurk that is hosted on another service such as SurveyGizmo you’ll want to start from the Survey Link template. That will let you post a task on MTurk that directs Workers to your survey.

On the first screen of this template you will be prompted to provide a name, description, and keywords for your survey. It’s always best to be descriptive as possible and include an estimate of how long your survey will take. In this case we’ve asked a few members of our team to complete our survey and it typically takes about 15 minutes. You can then specify how much you would like to pay Workers for each response and the number of Workers you want to participate in your survey. This is described as Assignments in MTurk. Here we are going to pay $3.00 per response and ask for 20 responses. To make sure that Workers have enough time to complete the survey, we allot them 30 minutes.

Make sure you note the Auto-Approval settings for your Project. This tells MTurk how long to wait before automatically approving a Worker’s response if you don’t do it yourself first. If you set this value to be too brief, it is possible that you may not leave yourself enough time to fully review Workers’ responses before they are automatically approved. If you set this value to be too long, Workers may become frustrated at the delay to receive their payment for completing your HIT. Customers performing surveys tend to find that 3 days is a good balance, but please adjust to suit your situation.

Adding your survey to the task layout

Your task layout describes what Workers will see when they choose to complete your survey. Here is a good place to include any special instructions. It’s also where you will provide a link to your survey. In SurveyGizmo select the Share tab and copy the Primary Share Link for your survey.

Now, replace the example link in your layout with the URL of your survey. The example is below:

After updating it:

You also need to update the link so that Workers can click on it. To do this, select the address and click on the link button.

You can then paste in your address:

When you get to the Preview step, be sure to check your link to make sure it brings up your survey when you click on it.

Tracking responses

If you’re using the Survey Link template and hosting your survey on another website such as SurveyGizmo, Qualtrics, or your own website, you’ll need to have some way to match responses to the Worker information in MTurk. You can do this by either providing Workers with a unique completion code or collecting a Worker’s ID in the survey itself. If your survey tool allows you to provide a unique completion code when complete you can then direct Workers to provide it in the MTurk task when they are done. You can then link the survey response to the Worker in your results. Instructions for doing this with SurveyGizmo are below.

The other option is to include a question in your survey that asks Workers for their Worker ID as part of the survey. Workers can generally find this on their Dashboard or in the upper left corner of the new Worker website. When you get your results, you’ll be able to match the Worker IDs. Note that it is a violation of MTurk policies to ask for a Worker’s email address.

Generating a Completion Code in SurveyGizmo

SurveyGizmo provides detailed information on how to create a completion code for your survey here. We’ve captured the relevant points but feel free to reference the original site if you need more information.

On any page before your Thank You page, select Add New Action and add a Hidden Value.

Give your new field a name of “Confirmation Code” and enter the following as the value to populate the field with:

[survey("counter"), safer="true", startat="100000"]

This will generate a sequence of unique values that should never repeat. Note that there is also an option to just insert a random number. However, because the values are truly random it’s possible for a number to repeat for two respondents. If you choose this option, make sure the range of values is very high which should limit the likelihood of a repeated value.

Edit your Thank You page to include a reference to the confirmation code and the code itself. To add the value to the page, click the Merge Code option in the editor toolbar to find the name of your Hidden Value in the menu. Select the name of your confirmation code and the value will be added to your page.

Test!

You’ve completed all the steps, but just to be safe before putting it in front of Workers, let’s try publishing it to just a few Workers first. This will give you a chance to take a look at the responses and make sure you’re getting what you’re looking for from your study. After all, nobody wants to spend hundreds or thousands of dollars on a study only to discover they made a minor mistake and will need to run it again.

Checking results

At any point after you publish your survey you can start checking your responses. If you hosted your survey entirely on MTurk, you can find all of your responses by viewing and downloading your Results on the Manage tab. If you hosted your survey elsewhere, you’ll need to download them from there and then match them up with your MTurk results. This can be done in Microsoft Excel or Google Sheets by using the VLOOKUP() function and matching on either a completion code or Worker ID.

VLOOKUP() allows you to look up a value in one worksheet from another worksheet. To get started, download your results as CSV (comma-seperated value) files from both MTurk and SurveyGizmo and open them in Excel. If you provided each Worker with a unique confirmation code that they entered in their MTurk task you will typically find it in column AB of your MTurk results and the WorkerID will be in column P. To link the WorkerID into your survey results file, first copy column P to column AE so that it’s to the right of the surveycode.

Now you can add a WorkerID column to your survey results. Add a new column heading called WorkerID and type =VLOOKUP( and you will be prompted to add four attributes. The first should be a cell in the same row in a column containing your survey code. The second should be a reference to the columns in your MTurk results file containing the surveycode and WorkerId, in this case they are AB:AE. The third value should be the number of columns between the surveycode and WorkerId including the columns themselves, in this case that’s 4. The final attribute should simply be FALSE. The result will look something like this:

=VLOOKUP(R2,Batch_2801044_batch_results.csv!$AB:$AE,4,FALSE)

Approving work

On the results tab in MTurk you’ll be able to see the responses for each Worker and Approve their work so that they get paid for completing it. Prompt approval is a great way to build trust with the Worker community and maintain a positive reputation.

Those are the steps to manage your survey responses with SurveyGizmo. We hope you found it useful.

If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.

--

--