Tutorial: Generating images with bounding boxes on-the-fly using AWS Lambda

Customers often use Amazon Mechanical Turk (MTurk) for image annotation tasks where Workers are asked to draw or verify bounding boxes over specific items or regions.

When working with such tasks, it can be useful to generate and save images with bounding box overlays. You might do this to visualize results from an MTurk Worker. Or to perhaps visualize a “ground truth” result that you plan to use to compare with the Worker’s response. Or you might overlay both together to quickly gauge the Worker’s accuracy. In this tutorial we will show you how to dynamically draw these bounding boxes on any image.

By the end of the tutorial, you will be able to embed image URLs in your task’s HTML to display images with overlaid bounding boxes at specified coordinates like so:

<!-- Original image without overlay -->
<img src='https://s3.amazonaws.com/mturk-solutions/tutorials/images/monkey.jpg'/>
<!-- Dynamically render a new version with a bounding box -->
<img src='https://jmwym8qv12.execute-api.us-east-1.amazonaws.com/Public?img=https%3A%2F%2Fs3.amazonaws.com%2Fmturk-solutions%2Ftutorials%2Fimages%2Fmonkey.jpg&left=375&top=300&width=225&height=200'/>

Displaying the two image links above will show us the result:

The URL to the bounding box image has 5 parameters you can configure -

img: a link to the original image on which you want to overlay bounding boxes, with special characters URL encoded

left, top: these two parameters define how far from the top and left edge of the image your bounding box should start. You can pass in multiple values for multiple bounding boxes as comma separated values. There is an example of that below.

width, height: these two parameters define the width and height of each of the bounding boxes you need

Here is an example of generating multiple bounding boxes on an image:

<!-- Original image without overlay -->
<img src='https://s3.amazonaws.com/mturk-solutions/tutorials/images/machines.jpg'/>
<!-- Dynamically render a new version with two bounding boxes -->
<img src='https://jmwym8qv12.execute-api.us-east-1.amazonaws.com/Public?img=https%3A%2F%2Fs3.amazonaws.com%2Fmturk-solutions%2Ftutorials%2Fimages%2Fmachines.jpg&left=140,700&top=210,210&width=300,310&height=530,550'/>

This adds two bounding boxes like so:

Here is how it works: the URL to the bounding box image is actually an AWS Gateway endpoint that is connected to an AWS Lambda function. When you send a request to the endpoint, it runs the function which loads the image from the img parameter, adds the bounding box overlay, stores the new image file in an AWS S3 bucket and sends back an HTTP 302 redirect to the S3 location. This makes the image with the overlay load automatically in the browser.

Now let’s build this!

You will need access to an AWS account. You can sign up for an account with free trial access.

Set up your S3 bucket
Your Lambda function needs a location where it can store the images it generates with bounding box overlays. We will use an S3 bucket to do so. To create an S3 bucket, visit the S3 Console in AWS and click on “Create Bucket”:

Set up the bucket with default settings and keep the name of the bucket handy for later.

Set up your Lambda function
Next, we still set up the actual Lambda function. We will be using Javascript and running our code using NodeJS.

First, let’s step through the actual Javascript code we will add to our Lambda function:

If you want to jump ahead, the full compiled Lambda function is ready to go here: https://s3.amazonaws.com/mturk-solutions/tutorials/boundingboxgenerator/TutorialCode.zip

Now lets create and configure the function. Go to the AWS Lambda console and click on the “Create function” button:

Select the “Author from scratch”:

Pick a name for your function, and ensure the runtime is set to Nodejs. Configure it to use a new Role (name it anything you like) with the standard “Basic Edge Lambda permissions” attached:

Click on “Create function” and once complete, you should see your new function set up as below:

We are going to trigger this function using an API Gateway endpoint. Click on “API Gateway” from the “Add triggers” menu on the left to set this up:

Scroll down the console to configure the API Gateway trigger. The API name and Deployment stage don’t matter too much. Change the Security field to “Open”:

This will allow anyone to access the URL which will help you test your workflow quickly. In production, you can control access but still allow a broad audience of MTurk Workers to call the endpoint using AWS Cognito. Here is a blog post that goes into further detail about how Cognito works with API Gateway.

After you have saved your changes, you should see your new endpoint set up:

Make note of the “Invoke URL” — that is the public endpoint you will be using.

Add S3 permissions to Role
The role that you created when setting up your function does not have permission to create files in your S3 bucket. We need to add those permissions. Go to the IAM console and click on “Roles” on the left hand column to find the role that was created earlier:

Click on the role and then select “Attach policy”:

Type in “S3” to filter the list that pops up and select “AmazonS3FullAccess”:

Now if you go back to the Lambda console and look at your function, you should see that it now has access to S3:

Before and after adding S3 permissions to your role

Now your function is configured for access! However, it doesn’t yet have the actual code we reviewed earlier. Let us update it to add that code now.

Update your Lambda code
We are going to update the function code by uploading a zip file that contains our code and the associated set of node modules that have been compiled for Amazon Linux.

Go to your function and scroll down to the Function code section. Select “Upload a file from Amazon S3”:

We have prepared a compiled zip file you can upload from: https://s3.amazonaws.com/mturk-solutions/tutorials/boundingboxgenerator/TutorialCode.zip. Enter that into the S3 link url field. Now is also a good time to create a Lambda Environment variable called “S3_BUCKET” and add the name of the S3 bucket you created earlier:

Be sure to change YOUR_S3_BUCKET to the name of the S3 bucket that you created.

One final change is to increase the Timeout on the Lambda function from 3 seconds to something higher. We have found that 6–7 seconds is generally sufficient but set it to something higher initially to ensure you don’t get timeout errors when testing:

Hit Save and your endpoint should be ready to use. Try calling it with an image and see if it works! You can now use your image generator when creating tasks on MTurk. If you are unfamiliar with how to do that, check out our basic primer tutorial.

You may run into errors initially. If you do, go to the Amazon Cloudwatch console and click on “Logs” to review the logs associated with your Lambda function to find errors:

Final remarks
A few final comments:

  1. This is a starting point. Our code has been simplified to keep it concise for the tutorial and thus has some obvious limitations: it does not support rotated bounding boxes and it does not have any checks in place to make sure that the image being passed in exists, or have any sanity checks on how many bounding boxes can be created and so on.
  2. To make changes to the code, download the zipped file we linked earlier, unzip it and modify the “index.js” file and then rezip and upload it to update your function.
  3. Our code makes use of the Sharp module to handle the image transformations. This module needs to be compiled to run on the AWS Lambda operating system (Amazon Linux). This can be done by generating the node_modules folder with “npm install sharp” on an Amazon Linux environment (such as an AWS EC2 instance, where you will need to first install gcc on using Yum). We completed this step for you in the zipped file above, but if you want to build this from scratch, this is something to be aware of.

We hope you find this tutorial useful. If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.