Important updates on MTurk marketplace integrity, Worker identity and Requester tools to manage task quality

Amazon Mechanical Turk (MTurk) has been connecting Workers from all over the world with Requesters to earn rewards for completing tasks anytime, anywhere for 15 years. We are committed to maintaining a healthy marketplace and we invest heavily in technology, processes, and systems to keep MTurk safe for both Workers and Requesters by actively attempting to prevent fraud and abuse. In addition, we want to help Workers get the work they want and Requesters the quality work they need.

Last year we focused heavily on improving marketplace health by investing in our fraud and abuse prevention program. This program is made up of a combination of systems and people that do the following: detect suspicious activity, take action against customers that violate our Participation Agreement (PA) and Acceptable Use Policy (AUP), and remove bad actors from the marketplace.

This program used a combination of advanced machine learning services and a team of highly trained investigators to find fraud and abuse across the marketplace. As you might imagine, we’re big fans of keeping humans in the loop, as a result we have found the people/machine combination highly effective at making the right decisions. For example, 99% of detected suspicious activity is reviewed by an investigator within 18 hours of it being flagged, and urgent feedback is reviewed within 1 hour. If we find that a Requester or a Worker has violated our PA or AUP, based on the seriousness of the violation, they may be warned or removed from the marketplace.

Workers and Requesters who feel their cases are unfairly adjudicated can appeal both warning and suspension decisions. While our customers may not always agree with our actions, we are confident that the investigation and enforcement process is fair. That said, we’re not perfect. With that in mind, if a Worker or Requester believes we took the wrong action, they can appeal that decision, which will initiate a separate investigation that’s reviewed with high priority. Rest assured that when someone tells us that we got it wrong, we take a second look. Unfortunately, some of our loudest detractors are users who have been suspended for committing fraud, violating the AUP, or other abuse of the marketplace. Our focus is keeping the marketplace safe for users acting in good faith.

Today we’re providing some new insights to this process. Some represent new mechanisms we’ve implemented in the past year, others have been in place for much longer. Our goal is to provide transparency where we’re able, and share how we think and act upon data that’s indicative of bad behavior.

To start, we leverage a variety of identity verification and user authentication systems to ensure that all users are validated. Here are a few examples:

• Amazon employs a robust identity validation process. Each US Worker completes a Tax Interview and provides official forms of identification to validate their identity which are reviewed individually prior to approval.
• Additionally, we use browser and client specific metadata to monitor Workers for suspicious activity, including but not limited to network location analysis and device fingerprinting to validate that people working on MTurk are where they claim to be.
• We use two step verification via phone number to help ensure each Worker account is unique.
• We run periodic audits for false, re-used, or cloned identities as an additional security measure and suspend inactive accounts to prevent account takeovers.

In 2019, we developed better processes to identify Workers who fail to exercise human intelligence and improved systems to identify Requesters who publish HITs that violate our PA or AUP. Our risk systems monitor account activity and escalate abnormal or suspicious user behavior. In addition to the signals generated from our own systems and models, we rely on our users to let us know when something isn’t right. We employ several methods designed to safeguard task quality and validate Worker identity. Here are a few of the ways we do this:

• We listen to Worker and Requester feedback through our customer support contacts, conversations with customers and social media. When patterns emerge in that feedback, we investigate and implement new policies to address emergent risks. Any report submitted by a Requester or Worker is reviewed by our investigations team to ensure appropriate action is taken in a timely manner.
• We identify and work with third parties as necessary to shut down websites where MTurk Worker accounts are changing hands, and move swiftly to remove fraudulent accounts from the marketplace. Anyone actively sharing accounts or trading in fraudulent accounts is in violation of the AUP and is immediately suspended upon discovery. When accounts are reported to us, we investigate and, when found to be fraudulent, remove those accounts from the marketplace.
• We run regular analysis on task completion attributes to ensure that Workers are adhering to the AUP and PA. Although we can’t identify every bad actor on the marketplace on day one, we remove many of these people very quickly.
• We work with researchers at several academic institutions to develop a third party view of MTurk marketplace health and use their insights to improve the user experience.
• When Workers report AUP violations, we immediately route that information to a member of our investigations team. Multiple reports on the same HIT or Requester expedites review. For example, HITs published by Requesters who neglect to use the Adult Content Qualification are removed from the marketplace. When Requesters receive more than one violation, their account is suspended.
• When Requesters make use of our Block and Rejection features, we pay attention. On the Requester side, if the Blocks and Rejections appear unusually high, we investigate to ensure workers are not being unfairly penalized. On the worker side, Workers with frequent Blocks and/or Rejections are routed for investigation and suspended if it is determined they are not using human intelligence to complete tasks.
• We identify Workers who are using scripts that autocomplete HITs and remove those users from the marketplace.

Customer Feedback is critical. Requesters can put safeguards in place and ensure the right Workers are selected for a task by leveraging the following features and functionality:

• Use Qualifications to select Workers with high approval ratings and enough task submissions to identify Workers with a history of high quality work.
• Reject low quality assignments to ensure that poor quality work is identified and not rewarded.
• Use the Block feature to prevent specific Workers from accepting tasks.

Worker feedback on Requester HITs helps us improve the quality of tasks available on the marketplace. Thank you for continually providing feedback on the HITs you encounter on MTurk.

• Report Broken HITs so that they can be removed from the marketplace by Requesters.
• Report HITs that violate the AUP or PA when you see them.

Both Requesters and Worker should feel empowered to contact us to report any issues not listed here at mturk.com/contact-us.

We appreciate all you do to protect the MTurk marketplace. We want MTurk users to know that we work every day to maintain a healthy marketplace where Workers and Requesters can connect. We are committed to partnering with our customers to ensure that our platform is safe and fair. In 2020, we’ll continue to build more robust identity management systems including identity validation for all international Workers to ensure that we have better information on international Workers. We also plan to expand geo-location options for Requesters so that we can support more tasks from non-US customers. This will help the marketplace continue to grow and provide more opportunities for Workers on MTurk. We made progress in 2019, but we know there is more work to do. We look forward to continued improvements in 2020.