Creating an API with AWS: Part 2: DynamoDB

In my post Creating an API with AWS, I demonstrated how to use a CloudFormation template to create a simple API with AWS. In this post, I’m going to walk through adding a DynamoDB database and POST method, and updating GET requests to pull data from the database.

Prerequisites

Before you start, you’ll need the following things:

1. An AWS account
2. An Administrator IAM User with which to use the AWS CLI
3. AWS CLI set up on your local machine
4. The CloudFormation template from my previous post, Creating an API with AWS

Note that AWS will charge you for creating and hosting this API if you don’t have Free Tier on your account.

Step 1: Add a DynamoDB Table to the CloudFormation Template

AWS DynamoDB is a fast, NoSQL database that is built around key-value pairs. For a simple web application, it is a good choice for data storage.

Taking the CloudFormation template (CFT) from my previous post, Creating an API with AWS, we can add a resource for a DynamoDB table which we’ll use to store the links under the Resources: line:

linkTable:
    Type: AWS::DynamoDB::Table
    Properties:
        AttributeDefinitions:
	    -
	        AttributeName: "link_id"
		AttributeType: "S"
	    -
		AttributeName: "link"
		AttributeType: "S"
	KeySchema:
	    -
		AttributeName: "link_id"
		KeyType: "HASH"
	    -
		AttributeName: "link"
		KeyType: "RANGE"
	ProvisionedThroughput:
	    ReadCapacityUnits: "5"
	    WriteCapacityUnits: "5"
	TableName: "linkTable"

This table will be called linkTable and have two attributes (fields): link_id, which will contain a unique id; and link, which will store the URL value. In the KeySchema section, we’ll define the link_id as the primary key and the link attribute as the sort key.

Step 2: Update the Lambda Execution Role in the CloudFormation Template

Next, we need to update the requestHandlerLambdaExecutionRole to enable the Lambda function to perform database operations. To do this, add the following policy underneath the existing policy named root:

-
    PolicyName: "db_access"
    PolicyDocument:
        Version: "2012-10-17"
        Statement:
            -
                Effect: "Allow"
                Action:
                    - dynamodb:DeleteItem
                    - dynamodb:GetItem
                    - dynamodb:PutItem
                    - dynamodb:Scan
                    - dynamodb:UpdateItem
               Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/*"

This policy will allow the Lambda function to delete, get, add, and update database table items as well as perform full table scans.

Step 3: Update the Lambda Function to use the DynamoDB Database

Replace the requestHandler section of the CFT with the following Lambda function, which will check if a request is a GET or a POST, and store or retrieve data to/from the database as required:

requestHandler:
    Type: "AWS::Lambda::Function"
    Properties: 
      Code:
        ZipFile: !Sub |
            import json
            import boto3
            import uuid

            # Establish credentials
            session_var = boto3.session.Session()
            credentials = session_var.get_credentials()

            """ --- Parameters --- """
            # DynamoDB tables, read lambda function environment variables or initialize with defaults if they do not exist.
            DYNAMODB_LINK_TABLE = 'linkTable'

            # Initialize DynamoDB Client
            dynamodb = boto3.client('dynamodb')

            def store_link(link):
                link_id = str(uuid.uuid4())
                dynamodb.put_item(
                    TableName=DYNAMODB_LINK_TABLE,
                    Item={
                        'link_id': {
                            'S': str(link_id)
                        },
                        'link': {
                            'S': str(link)
                        }
                    }
                )

                return link_id
                    
            def get_links():
                links = []
                result = dynamodb.scan(
                            TableName=DYNAMODB_LINK_TABLE,
                            Select="SPECIFIC_ATTRIBUTES",
                            ProjectionExpression="link_id,link"
                        )
                        
                if result['Count'] > 0:
                    for x in range(result['Count']):
                        link = {}
                        link['link_id'] = result['Items'][x]['link_id']['S']
                        link['link'] = result['Items'][x]['link']['S']
                        links.append(link)
                            
                return links
                        
                    
            def error_response(error_message, aws_request_id):
                    return {
                        "statusCode": 500,
                        "isBase64Encoded": False,
                        "body": json.dumps({
                            "Error": error_message,
                            "Reference": aws_request_id,
                        }),
                        "headers": {
                            'Access-Control-Allow-Origin': '*',
                        },
                    }

            def lambda_handler(event, context):
                try:
                    print(event)
                    print(context)
                    return_body = ""
                    if event['httpMethod'] == "POST":
                        post_body = json.loads(event['body'])
                        link = post_body['link']
                        link_id = store_link(link)
                        return_body = json.dumps({"link_id": link_id})
                    elif event['httpMethod'] == "GET":
                        links = get_links()
                        return_body = json.dumps(links)
                    # Because this Lambda function is called by an API Gateway proxy integration
                    # the result object must use the following structure.
                    return {
                        "statusCode": 200,
                        "isBase64Encoded": False,
                        "body": return_body,
                        "headers": {
                                'Access-Control-Allow-Origin': '*',
                            },
                    }
                except Exception as err:
                    print(str(err))

                    # If there is an error during processing, catch it and return
                    # from the Lambda function successfully. Specify a 500 HTTP status
                    # code and provide an error message in the body. This will provide a
                    # more meaningful error response to the end client.
                    error_response(str(err), context.aws_request_id)

      Description: Lambda function for processing api endpoint requests
      FunctionName: requestHandler
      Handler: index.lambda_handler
      Role: 
        Fn::GetAtt: 
        - "requestHandlerLambdaExecutionRole"
        - "Arn"
      Runtime: python3.8
    DependsOn:
        - requestHandlerLambdaExecutionRole

Step 4: Add a POST Method to the API in the CloudFormation Template

Add a POST method to the API so that new links can be added. To do this, add another AWS::APIGateway::Method resource under ProxyResourceGET called ProxyResourcePOST:

ProxyResourcePOST:
    Type: AWS::ApiGateway::Method
    Properties:
        RestApiId: !Ref ApplicationAPI
        ResourceId: !Ref link
        HttpMethod: POST   
        AuthorizationType: NONE
        Integration:
            Type: AWS_PROXY
            IntegrationHttpMethod: POST
            Uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${requestHandler.Arn}/invocations
            IntegrationResponses:
                -
                    StatusCode: 200
        MethodResponses:
            - 
                StatusCode: 200
                ResponseModels:
                    application/json: Empty
    DependsOn: 
        - ApplicationAPI

Step 5: Launch the CloudFormation Template

Update the Description field to something that better describes what this CFT is building, e.g. “Creates a simple API endpoint that can be called to POST a link and store it in a database, or GET a list of links”. Save the CFT to a file called setupapidb.yml. The complete CFT should now look as below:

AWSTemplateFormatVersion: "2010-09-09"
Description: Creates a simple API endpoint that can be called to POST a link and store it in a database, or GET a list of links
Resources:
    linkTable:
        Type: AWS::DynamoDB::Table
        Properties:
          AttributeDefinitions:
            -
              AttributeName: "link_id"
              AttributeType: "S"
            -
              AttributeName: "link"
              AttributeType: "S"
          KeySchema:
            -
              AttributeName: "link_id"
              KeyType: "HASH"
            -
              AttributeName: "link"
              KeyType: "RANGE"
          ProvisionedThroughput:
            ReadCapacityUnits: "5"
            WriteCapacityUnits: "5"
          TableName: "linkTable"
                
    requestHandlerLambdaExecutionRole:
        Type: AWS::IAM::Role
        Properties: 
          AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
                - 
                    Effect: Allow
                    Principal:
                        Service:
                            - "lambda.amazonaws.com"
                    Action:
                        - "sts:AssumeRole"
          Path: "/"
          Policies:
            - 
                PolicyName: "root"
                PolicyDocument:
                    Version: "2012-10-17"
                    Statement:
                        - 
                            Effect: "Allow"
                            Action:
                                - logs:CreateLogGroup
                                - logs:PutLogEvents
                                - logs:CreateLogStream
                            Resource: "*"
            -
                PolicyName: "db_access"
                PolicyDocument:
                    Version: "2012-10-17"
                    Statement:
                        -
                            Effect: "Allow"
                            Action:
                                - dynamodb:DeleteItem
                                - dynamodb:GetItem
                                - dynamodb:PutItem
                                - dynamodb:Scan
                                - dynamodb:UpdateItem
                            Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/*"
          RoleName: requestHandlerLambdaExecutionRole


    requestHandler:
        Type: "AWS::Lambda::Function"
        Properties: 
          Code:
            ZipFile: !Sub |
                import json
                import boto3
                import uuid

                # Establish credentials
                session_var = boto3.session.Session()
                credentials = session_var.get_credentials()

                """ --- Parameters --- """
                # DynamoDB tables, read lambda function environment variables or initialize with defaults if they do not exist.
                DYNAMODB_LINK_TABLE = 'linkTable'

                # Initialize DynamoDB Client
                dynamodb = boto3.client('dynamodb')

                def store_link(link):
                    link_id = str(uuid.uuid4())
                    dynamodb.put_item(
                        TableName=DYNAMODB_LINK_TABLE,
                        Item={
                            'link_id': {
                                'S': str(link_id)
                            },
                            'link': {
                                'S': str(link)
                            }
                        }
                    )

                    return link_id
                    
                def get_links():
                    links = []
                    result = dynamodb.scan(
                                TableName=DYNAMODB_LINK_TABLE,
                                Select="SPECIFIC_ATTRIBUTES",
                                ProjectionExpression="link_id,link"
                            )
                        
                    if result['Count'] > 0:
                        for x in range(result['Count']):
                            link = {}
                            link['link_id'] = result['Items'][x]['link_id']['S']
                            link['link'] = result['Items'][x]['link']['S']
                            links.append(link)
                            
                    return links
                        
                    
                def error_response(error_message, aws_request_id):
                        return {
                            "statusCode": 500,
                            "isBase64Encoded": False,
                            "body": json.dumps({
                                "Error": error_message,
                                "Reference": aws_request_id,
                            }),
                            "headers": {
                                'Access-Control-Allow-Origin': '*',
                            },
                        }

                def lambda_handler(event, context):
                    try:
                        print(event)
                        print(context)
                        return_body = ""
                        if event['httpMethod'] == "POST":
                            post_body = json.loads(event['body'])
                            link = post_body['link']
                            link_id = store_link(link)
                            return_body = json.dumps({"link_id": link_id})
                        elif event['httpMethod'] == "GET":
                            links = get_links()
                            return_body = json.dumps(links)
                        # Because this Lambda function is called by an API Gateway proxy integration
                        # the result object must use the following structure.
                        return {
                            "statusCode": 200,
                            "isBase64Encoded": False,
                            "body": return_body,
                            "headers": {
                                    'Access-Control-Allow-Origin': '*',
                                },
                        }
                    except Exception as err:
                        print(str(err))

                        # If there is an error during processing, catch it and return
                        # from the Lambda function successfully. Specify a 500 HTTP status
                        # code and provide an error message in the body. This will provide a
                        # more meaningful error response to the end client.
                        error_response(str(err), context.aws_request_id)

          Description: Lambda function for processing api endpoint requests
          FunctionName: requestHandler
          Handler: index.lambda_handler
          Role: 
            Fn::GetAtt: 
            - "requestHandlerLambdaExecutionRole"
            - "Arn"
          Runtime: python3.8
        DependsOn:
            - requestHandlerLambdaExecutionRole
            
    ApplicationAPI:
        Type: AWS::ApiGateway::RestApi
        Properties:
            Name: ApplicationAPI
            Description: Simple API for an application
        DependsOn: 
            - requestHandler
            
    LambdaPermission:
        Type: AWS::Lambda::Permission
        Properties:
            FunctionName: !GetAtt 
              - requestHandler
              - Arn
            Action: 'lambda:InvokeFunction'
            Principal: apigateway.amazonaws.com
            SourceArn: !Join ["", [!Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:", !Ref ApplicationAPI, "/*"]]
        DependsOn:
            - ApplicationAPI
            
    link:
        Type: AWS::ApiGateway::Resource
        Properties:
            RestApiId: !Ref ApplicationAPI
            ParentId: !GetAtt [ApplicationAPI, RootResourceId]
            PathPart: 'link'
        DependsOn: 
            - ApplicationAPI
    
    ProxyResourceGET:
        Type: AWS::ApiGateway::Method
        Properties:
            RestApiId: !Ref ApplicationAPI
            ResourceId: !Ref link
            HttpMethod: GET     
            AuthorizationType: NONE
            Integration:
                Type: AWS_PROXY
                IntegrationHttpMethod: POST
                Uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${requestHandler.Arn}/invocations
                IntegrationResponses:
                    -
                        StatusCode: 200
            MethodResponses:
                - 
                    StatusCode: 200
                    ResponseModels:
                        application/json: Empty
        DependsOn: 
            - ApplicationAPI
            
    ProxyResourcePOST:
        Type: AWS::ApiGateway::Method
        Properties:
            RestApiId: !Ref ApplicationAPI
            ResourceId: !Ref link
            HttpMethod: POST   
            AuthorizationType: NONE
            Integration:
                Type: AWS_PROXY
                IntegrationHttpMethod: POST
                Uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${requestHandler.Arn}/invocations
                IntegrationResponses:
                    -
                        StatusCode: 200
            MethodResponses:
                - 
                    StatusCode: 200
                    ResponseModels:
                        application/json: Empty
        DependsOn: 
            - ApplicationAPI
              
    OptionsMethod:
        Type: AWS::ApiGateway::Method
        Properties:
            AuthorizationType: NONE
            RestApiId: !Ref ApplicationAPI
            ResourceId: !Ref link
            HttpMethod: OPTIONS
            Integration:
                IntegrationResponses:
                    - 
                        StatusCode: 200
                        ResponseParameters:
                            method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
                            method.response.header.Access-Control-Allow-Methods: "'GET,POST,OPTIONS'"
                            method.response.header.Access-Control-Allow-Origin: "'*'"
                        ResponseTemplates:
                            application/json: ''
                PassthroughBehavior: NEVER
                RequestTemplates:
                    application/json: '{"statusCode": 200}'
                Type: MOCK
            MethodResponses:
                - 
                    StatusCode: 200
                    ResponseModels:
                        application/json: Empty
                    ResponseParameters:
                        method.response.header.Access-Control-Allow-Headers: true
                        method.response.header.Access-Control-Allow-Methods: true
                        method.response.header.Access-Control-Allow-Origin: true
        DependsOn: 
            - ProxyResourceGET
                
    Deployment: 
      DependsOn: "OptionsMethod"
      Type: AWS::ApiGateway::Deployment
      Properties: 
        RestApiId: 
          Ref: "ApplicationAPI"
        Description: "Application API deployment"
        StageName: "prod"
Outputs:
  APIUrl:
    Description: Application API Gateway link endpoint URL
    Value: !Join
        - ''
        - - 'https://'
          - !Ref ApplicationAPI
          - '.execute-api.'
          - !Ref AWS::Region
          - '.amazonaws.com/prod/link'

Launch the CFT with the following command:

$ aws cloudformation create-stack --stack-name apidb --template-body file://setupapidb.yml --capabilities CAPABILITY_NAMED_IAM

Poll the progress of the CloudFormation stack with the following command:

$ aws cloudformation describe-stacks --stack-name apidb

Once complete, the value of “StackStatus” in the response should be “CREATE_COMPLETE” and you should have the API endpoint link in the “Outputs” section, under “OutputValue”. The link should be of the form: “https://xxxxxxxxxx.execute-api.your-region.amazonaws.com/prod/link”.

Step 6: Test the API

Test that everything is working by using curl. You may need to install curl first with sudo yum install curl or similar, depending on your operating system. Once installed, issue a curl POST request like the following. Be sure to replace the endpoint url with your own:

$ curl -v -X "POST" -H "Content-Type: application/json" -d "{\"link\": \"https://www.google.com\"}" https://xxxxxxxxxx.execute-api.your-region.amazonaws.com/prod/link

If the request is successful, you should see a response with a lot of connection information, and at the end of it, a line similar to this one:

{"link_id": "177173df-0e92-41a9-b2df-e8722443fd8a"}

That is the link_id of the link database entry just created.

Now try a GET request with the following curl request:

$ curl https://xxxxxxxxxx.execute-api.your-region.amazonaws.com/prod/link

If all is working, you should get a response like the below:

$ [{"link_id": "177173df-0e92-41a9-b2df-e8722443fd8a", "link": "https://www.google.com"}]

Try POSTing different links and then calling GET again. You should see the list of links getting larger.

Step 7: Clean Up

If you don’t want to keep your API (note that AWS will charge you for it if you don’t have Free Tier on your account), delete the CFT stack (all resources created) by running:

$ aws cloudformation delete-stack --stack-name apidb

You will then need to remove any logs that were generated by the Lambda function. To do this:

1. Log into the AWS Console (your AWS account) using your Administrator IAM user, and navigate to CloudWatch.
2. On the left-hand side, select Logs -> Log Groups.
3. Select ‘aws/lambda/requestHandler’.
4. Under Actions, select ‘Delete log groups’ and confirm.

Conclusion

You should now be able to create a simple API on AWS with a CloudFormation template, that stores and retrieves data to and from a DynamoDB database. If you’re interested in adding additional endpoints and databases, see my follow-on post, Creating an API with AWS: Part 3: Additional Endpoints.

Thanks for reading!