Understanding HIT States
Amazon Mechanical Turk (MTurk) is a marketplace for human tasks known as Human Intelligence Tasks (HITs) such as those to generate training data for machine learning. Customers who complete HITs are called Workers and customers who publish these tasks are called Requesters.
Requesters can use the MTurk Requester Application Programming Interface (API) to automate the submission of HITs and collection of their results.
When working with the API, it is useful to understand the “state” of a HIT as it will help you keep track of the progress of your tasks. State is a flag that can be read and modified using the API.
This post will walk through the different HIT states available and how to work with them.
HITs can exist in one of five states:
- Assignable — a HIT can be accepted and worked on by Workers
- Unassignable — a HIT cannot be worked on by any more Workers
- Reviewable — all Workers have submitted answers to a HIT
- Reviewing (optional state) — the HIT has been marked as “Reviewing” by the Requester
- Disposed — the HIT has been deleted and can no longer be retrieved
Remember, when a Worker completes your HIT, you get back an Assignment containing their results. You can ask for multiple assignments for a HIT using the MaxAssignments parameter when calling the CreateHIT API operation. This means that multiple Workers will complete the same work for you. This can be useful to cross-check the accuracy of work or get different perspectives on the same question from multiple Workers.
Once HITs are created, they are in an Assignable State
Every HIT is created in the Assignable State when you call the CreateHit API operation. Once a HIT is Assignable it means that Workers can accept it and start working on it.
HITs typically go from Assignable to Unassignable
HITs become Unassignable when either the last assignment for that HIT is accepted by a Worker or when the HIT expires. HITs expire when the time allotted for the HIT passes. This is something you define with the LifetimeInSeconds parameter in your CreateHIT call. You can also manually expire a HIT by calling the UpdateExpirationForHIT API operation.
A HIT will return to the Assignable state if a Worker returns a HIT assignment or if you add assignments to the HIT using the CreateAdditionalAssignmentsForHIT API call.
HITs become Reviewable when all assignments are submitted
Once Workers have submitted all the assignments for a HIT, it goes into the Reviewable state.
The Reviewable state can be used in two ways:
- You can use the ListReviewableHITs operation to get a list of all HITs that are in the Reviewable state. This is a good way to retrieve results from MTurk for work that has been completed.
- You can get an Amazon SNS or SQS push notification from MTurk when a HIT becomes Reviewable. We published a tutorial of how to use SNS with MTurk last week. This is a handy way to create automated workflows to download results from MTurk as soon as they are ready.
Approving or rejecting assignments from Workers does not affect the Reviewable state of a HIT. However adding more assignments to a HIT will revert it back to the Assignable state.
Optional Reviewing State
The Reviewing state can be set manually using the UpdateHITReviewStatus API call. HITs in the Reviewing state will not be returned when calling ListReviewableHITs. This can be useful as you can use this state to filter out HITs you have already checked and use ListReviewableHITs to only pull new Reviewable HITs.
Calling the DeleteHIT API operation sets the HIT state to Disposed. Deleted HITs can no longer be accessed from MTurk. HITs are automatically deleted when they have been inactive for 120 days.
Note that a HIT can only be deleted when it is in the Reviewable state and all its assignments have been approved or rejected. If you want to take down a HIT that is currently “live” and available to Workers (that is, Assignable), then call the UpdateExpirationForHIT operation and set the expiration time to 0.
We hope you find this tutorial useful. If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.