What is in the picture with AWS Rekognition

AWS Rekognition offers an out of the box experience for image or video recognition. The base model works well for everyday objects. Beyond that, one can use Custom Labels and train their model for recognising objects beyond the base model’s scope.

One can see Rekognition capability on everyday pictures on my Twitter timeline:

Using Rekognition with Python is simple as it’s part of boto3 client and one invokes a single detect_labels method.

import boto3

aws_access_key = 'xxx'
aws_key_id = 'xxx'
file_name = "biergarten.jpeg"

client = boto3.client(
    "rekognition",
    aws_access_key_id=aws_access_key,
    aws_secret_access_key=aws_key_id,
    region_name="eu-central-1",
)

with open(file_name, "rb") as photo:
    photo_bytes = photo.read()

response = client.detect_labels(Image={"Bytes": photo_bytes})
tags = [
    f"{round(t['Confidence'],0):.0f}% #{t['Name'].replace(' ', '')}"
    for t in response["Labels"]
]
print(f"Rekognition detected the following labels: {tags}")

print(response)

In the response we get a Python dictionary with detected labels:

{'Labels': [{'Name': 'Bread', 'Confidence': 99.53875732421875, 'Instances': [], 'Parents': [{'Name': 'Food'}]}, {'Name': 'Food', 'Confidence': 99.53875732421875, 'Instances': [], 'Parents': []}, {'Name': 'Cracker', 'Confidence': 97.35413360595703, 'Instances': [], 'Parents': [{'Name': 'Bread'}, {'Name': 'Food'}]}, {'Name': 'Pretzel', 'Confidence': 85.56755828857422, 'Instances': [], 'Parents': [{'Name': 'Cracker'}, {'Name': 'Bread'}, {'Name': 'Food'}]}, {'Name': 'Hot Dog', 'Confidence': 61.456153869628906, 'Instances': [{'BoundingBox': {'Width': 0.5234535336494446, 'Height': 0.6410773992538452, 'Left': 0.31120765209198, 'Top': 0.24137811362743378}, 'Confidence': 61.456153869628906}], 'Parents': [{'Name': 'Food'}]}, {'Name': 'Dish', 'Confidence': 56.22080612182617, 'Instances': [], 'Parents': [{'Name': 'Meal'}, {'Name': 'Food'}]}, {'Name': 'Meal', 'Confidence': 56.22080612182617, 'Instances': [], 'Parents': [{'Name': 'Food'}]}], 'LabelModelVersion': '2.0', 'ResponseMetadata': {'RequestId': 'a30e823d-67e4-442a-9e63-e262dab4c7e1', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1', 'date': 'Fri, 29 Jan 2021 12:29:42 GMT', 'x-amzn-requestid': 'a30e823d-67e4-442a-9e63-e262dab4c7e1', 'content-length': '887', 'connection': 'keep-alive'}, 'RetryAttempts': 0}}