Tutorial: Getting great survey results from MTurk and Google Forms
MTurk is a great way to get fast responses to your survey or study but it works a little differently from traditional survey services. Today, we’re going to talk about how to manage the results of surveys you host on Google Forms. You can also host your survey on MTurk itself but some customers choose to use Google Forms, Qualtrics, or a number of other survey tools they may be more familiar with. To learn how to use MTurk with other survey services, check out these past posts about Qualtrics and Survey Monkey.
Getting Started
To post a survey on MTurk that is hosted on another service such as Google Forms you’ll want to start from the Survey Link template. That will let you post a task on MTurk that directs Workers to your survey.
On the first screen of this template you will be prompted to provide a name, description, and keywords for your survey. It’s always best to be descriptive as possible and include an estimate of how long your survey will take. In this case we’ve asked a few members of our team to complete our survey and it typically takes about 15 minutes. You can then specify how much you would like to pay Workers for each response and the number of Workers you want to participate in your survey. This is described as Assignments in MTurk. Here we are going to pay $3.00 per response and ask for 20 responses. To make sure that Workers have enough time to complete the survey, we allot them 30 minutes.
Make sure you note the Auto-Approval settings for your Project. This tells MTurk how long to wait before automatically approving a Worker’s response if you don’t do it yourself first. If you set this value to be too brief, it is possible that you may not leave yourself enough time to fully review Workers’ responses before they are automatically approved. If you set this value to be too long, Workers may become frustrated at the delay to receive their payment for completing your HIT. Customers performing surveys tend to find that 3 days is a good balance, but please adjust to suit your situation.
Adding your survey to the task layout
Your task layout describes what Workers will see when they choose to complete your survey. Here is a good place to include any special instructions. It’s also where you will provide a link to your survey. In Google Forms select the Send button in the upper right and choose the Link option (it’s the middle one).
Now, replace the example link in your layout with the URL of your survey. The example is below:
After updating it:
You also need to update the link so that Workers can click on it. Do do this select the address and click on the link button.
You can then paste in your address:
When you get to the Preview step, be sure to check your link to make sure it brings up your survey when you click on it.
Tracking responses
If you’re using the Survey Link template and hosting your survey on another website such as Google Forms, Qualtrics, or your own website, you’ll want to have some way to validate that Workers that submit tasks on MTurk completed your survey. You can do this by either by providing Workers with a completion code or collecting a Worker’s ID in the survey itself and matching them up. If your survey tool allows you to provide a completion code when complete you can then direct Workers to provide it in the MTurk task when they are done. You can then link the survey response to the Worker in your results.
Google Forms doesn’t have a method to provide a unique completion code so the best option is include a question in your survey that asks Workers for their Worker ID. Workers can generally find this on their Dashboard or in the upper left corner of the new Worker website. When you get your results you’ll be able to match the Worker IDs. Note that it is a violation of MTurk policies to ask for a Worker’s email address.
Note that the Survey Link template includes a field where Workers are asked to provide a completion code. Don’t forget to add this to your survey! Simply select settings in Google Forms and change the Presentation settings to include a completion message. Feel free to pick your own message and code.
Test!
You’ve completed all the steps, but just to be safe before putting it in front of Workers, let’s try publishing it to just a few Workers first. This will give you a chance to take a look at the responses and make sure you’re getting what you’re looking for from your study. After all, nobody wants to spend hundreds or thousands of dollars on a study only to discover they made minor mistake and will need to run it again.
Checking results
At any point after you publish your survey you can start checking your responses. If you hosted your survey entirely on MTurk, you can find all of your responses by viewing and downloading your Results on the Manage tab. If you hosted your survey elsewhere, you’ll need to download them from there and then match them up with your MTurk results. This can be done in Microsoft Excel or Google Sheets by using the VLOOKUP() function and matching on Worker ID.
VLOOKUP() allows you to look up a value in one worksheet from another worksheet. To get started, download your results as CSV (comma-seperated value) files from both MTurk and Google Forms and open them in Excel. If you asked for a completion code you will typically find it in column AB of your MTurk results and the WorkerID will be in column P. To link the WorkerID into your survey results file, first copy column P to column AE so that it’s the the right of the surveycode.
Now you can add a WorkerID column to your survey results. Add a new column heading called WorkerID and type =VLOOKUP( and you will be prompted to add four attributes. The first should be a cell in the same row in a column containing your survey code. The second should be a reference to the columns in your MTurk results file containing the surveycode and WorkerId, it this case they are AB:AE. The third value should be the number of columns between the surveycode and WorkerId including the columns themselves, in this case that’s 4. The final attribute should simply be FALSE. The result will look something like this:
=VLOOKUP(R2,Batch_2801044_batch_results.csv!$AB:$AE,4,FALSE)
Approving work
On the results tab in MTurk you’ll be able to see the responses for each Worker and Approve their work so that they get paid for completing it. Prompt approval is a great way to build trust with the Worker community and maintain a positive reputation.
Those are the steps to manage your survey responses with Google Forms. We hope you found it useful.
If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.
