Serverless Delivery: Bootstrapping (Part 2 of 3)

In the first of this three part series on Serverless Delivery, we took a look at the high level architecture of running a continuous delivery pipeline with CodePipeline + Lambda. Our objective is to run the Dromedary application in a serverless environment with a serverless continuous delivery pipeline.
Before we can build the pipeline, we need to have the platform in place to deploy our application to. In this second part of the series we will look at what changes need to be made to a Node.js Express application to run in Lambda and the CloudFormation templates needed to create the serverless resources to host our application.

Prepare Your Application

Lambdas are defined as a function that takes in an event object containing data elements mapped to it in the API Gateway Integration Request. An application using Express in Node.js however expects its request to be initiated from an HTTP request on a socket. In order to run your Express application as a Lambda function, you’ll need some code to mediate between the two frameworks.
Although the best approach would be to have your application natively support the Lambda event, this may not always be feasible. Therefore, I have created a small piece of code to serve as a mediator and put it outside of the Dromedary application in its own module named lambda-express for others to leverage.

Install the module with npm install –save lambda-express and then use it in your Express application to define the Lambda handler:

var lambdaExpress = require('lambda-express');
var express = require('express');
var app = express();
// ... initialize the app as usual ...
// create a handler function that maps lambda inputs to express
exports.handler = lambdaExpress.appHandler(app);

In the dromedary application, this is available in a separate index.js file. You’ll also notice in the dromedary application, that it passes a callback function rather than the express app to the appHandler function. This allows it to use information on the event to configure the application, in this case via environment variables:

exports.handler = lambdaExpress.appHandler(function(event,context) {
process.env.DROMEDARY_DDB_TABLE_NAME = event.ddbTableName;
var app = require('./app.js');
return app;
});

You now have an Express application that will be able to respond to Lambda events that are generated from API Gateway. Now let’s look at what resources need to be defined in your CloudFormation templates. For the dromedary application, these templates are defined in a separate repository named dromedary-serverless in the pipeline/cfn directory.

Define Website Resources

Sample – site.json
Buckets need to be defined for test and production stages of the pipeline to host the static content of the application. This includes the HTML, CSS, images and any JavaScript that will run in the browser. Each bucket will need a resource like what you see below.

"TestBucket" : {
  "Type" : "AWS::S3::Bucket",
  "Properties" : {
    "AccessControl" : "PublicRead",
    "BucketName" : “www-test.mysite.com”,
    "WebsiteConfiguration" : {
      "IndexDocument" : "index.html"
    }
  }
}

Here are the important pieces to pay attention to:

AccessControl – set to to PublicRead assuming this is a public website.
WebsiteConfiguration – add an IndexDocument entry to define the home page for the bucket.
BucketName – needs to match exactly the Name for the Route53 ResourceRecord you create. For example, if I’m going to setup a DNS record for www-test.mysite.com , then the bucket name should also be www-test.mysite.com .

We will also want Route53 resource records for each of the test and production buckets. Here is a sample record:

"TestSiteRecord": {
  "Type": "AWS::Route53::RecordSetGroup",
  "Properties": {
    "HostedZoneId": “Z00ABC123DEF”,
    "RecordSets": [{
      "Name": “www-test.mysite.com.”,
      "Type": "A",
      "AliasTarget": {
        "HostedZoneId": “Z3BJ6K6RIION7M”,
        "DNSName": “s3-website-us-west-2.amazonaws.com"
      }
    }]
  }
}

Make sure that your record does the following:

Name – must be the same as the bucket name above with a period at the end.
HostedZoneId – should be specific to your account and for the zone you are hosting (mysite.com in this example).
AliasTarget – will be reference the zone id and endpoint for S3 in the region you created the bucket. The zone ids and endpoints can be found in the AWS General Reference Guide.

Declare Lambda Functions

Sample – app.json
Lambda functions will need to be declared for test and production stages of the pipeline to serve the Express application. Each function will only be stubbed out in the CloudFormation template so the API Gateway resources can reference it. Each execution of a CodePipeline job will deploy the latest version of the code as a new version of the function. Here is a sample declaration of the Lambda function in CloudFormation:

"TestAppLambda": {
  "Type" : "AWS::Lambda::Function",
  "Properties" : {
    "Code" : {
      "ZipFile": { "Fn::Join": ["n", [
        "exports.handler = function(event, context) {",
        " context.fail(new Error(500));",
        "};"
      ]]}
    },
    "Description" : "serverless application",
    "Handler" : "index.handler",
    "MemorySize" : 384,
    "Timeout" : 10,
    "Role" : {“Ref”: “TestAppLambdaTrustRole”},
    "Runtime" : "nodejs"
  }
}

Notice the following about the Lambda resource:

ZipFile – the default implementation of the function is provided inline. Notice it just returns a 500 error. This will be replaced by real code when CodePipeline runs.
MemorySize – this is the only control you have over the system resources allocated to your function. CPU performance is determined by the amount of memory you allocate, so if you need more CPU, increase this number. Your cost is directly related to this number as is the duration of each invocation. There is a sweet spot you need to find where you get the shortest durations for the system resources.
Timeout – max time (in seconds) for a given invocation of the function to run before it is forcibly terminated. The maximum value for this is 300 seconds.
Role – reference the ARN of the IAM role that you want to assign to your function when it runs. You’ll want to have “Principal”:{“Service”:[“lambda.amazonaws.com”]} in the AssumePolicyDocument to grant the Lambda service access to the sts:AssumeRole action. You’ll also want to include in the policy access to CloudWatch Logs with “Action”: [“logs:CreateLogGroup”,”logs:CreateLogStream”,”logs:PutLogEvents”]

Define API Gateway and Stages

Sample – api-gateway.json
Our Lambda function requires something to receive HTTP requests and deliver them as Lambda events. Fortunately, the AWS API Gateway is a perfect solution for this need. A single API Gateway definition with two stages, one for test and one for production will be defined to provide the public access to your Lambda function defined above. Unfortunately, CloudFormation does not have support yet for API Gateway. However, Andrew Templeton has created a set of custom resources that do a great job of filling the gap. For each package, you will need to create a Lambda function in your CloudFormation template, for example:

"ApiGatewayRestApi": {
  "Type" : "AWS::Lambda::Function",
  "Properties" : {
    "Code" : {
      "S3Bucket": “dromedary-serverless-templates”,
      "S3Key": "cfn-api-gateway-restapi.zip"
    },
    "Description" : "Custom CFN Lambda",
    "Handler" : "index.handler",
    "MemorySize" : 128,
    "Timeout" : 30,
    "Role" : { "Ref": "ApiGatewayCfnLambdaRole" },
    "Runtime" : "nodejs"
  }
}

Make sure the role that you create and reference from the Lambda above contains policy statements to give it access to apigateway:* actions, as well as granting the custom resource access to PassRole the role defined for API Integration.

{
  "Effect": "Allow",
  "Resource": [
    { "Fn::GetAtt": [ "ApiIntegrationCredentialsRole", "Arn" ] }
  ],
  "Action": [
    "iam:PassRole"
  ]
}

Defining the API Gateway above consists of the following six resources. For each one, I will highlight only the important properties to be aware of as well as a picture from the console of what the CloudFormation resource creates:

cfn-api-gateway-restapi – this is the top level API definition.
Name – although not required, you ought to provide a unique name for the API

cfn-api-gateway-resource – a resource (or path) for the API. In the reference application, I’ve created just one root resource that is a wildcard and captures all sub paths. The sub path is then passed into lambda-express as a parameter and mapped into a path that Express can handle.

PathPart – define a specific path of your API, or {subpath} to capture all paths as a variable named subpath
ParentId – This must reference the RootResourceId from the restapi resource
```
"ParentId": { "Fn::GetAtt": [ "RestApi", "RootResourceId" ] }
```

cfn-api-gateway-method – defines the contract for a request to an HTTP method (e.g., GET or POST) on the path created above.

HttpMethod – the method to support (GET)
RequestParameters – a map of parameters on the request to expect and pass down to the integration
```
"RequestParameters": {
  "method.request.path.subpath": true
}
```

cfn-api-gateway-method-response – defines the contract for the response from an HTTP method defined above.

HttpMethod – the method to support
StatusCode – the code to return for this response (e.g., 200)

ResponseParameters – a map of the headers to include in the response

"ResponseParameters": {
  "method.response.header.Access-Control-Allow-Origin": true,
  "method.response.header.Access-Control-Allow-Methods": true,
  "method.response.header.Content-Type": true
}

cfn-api-gateway-integration – defines where to send the request for a given method defined above.

Type – for Lambda function integration, choose AWS
IntegrationHttpMethod – for Lambda function integration, choose POST
Uri – the AWS service URI to integrate with. For Lambda use the example below. Notice that the function name and version are not in the URI, but rather there are variables in their place. This way we can allow the different stages (test and production) to control which Lambda function and which version of that function to call:
```
arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2::function:${stageVariables.AppFunctionName}:${stageVariables.AppVersion}/invocations
```
Credentials – The role to run as when invoking the Lambda function
RequestParameters – The mapping of parameters from the request to integration request parameters
```
"RequestParameters": {
  "integration.request.path.subpath": "method.request.path.subpath"
}
```

RequestTemplates – The template for the JSON to pass to the Lambda function. This template captures all the context information from API Gateway that lambda-express will need to create the request that Express understands:

"RequestTemplates": {
  "application/json": {
    "Fn::Join": ["n",[
      "{",
      " "stage": "$context.stage",",
      " "request-id": "$context.requestId",",
      " "api-id": "$context.apiId",",
      " "resource-path": "$context.resourcePath",",
      " "resource-id": "$context.resourceId",",
      " "http-method": "$context.httpMethod",",
      " "source-ip": "$context.identity.sourceIp",",
      " "user-agent": "$context.identity.userAgent",",
      " "account-id": "$context.identity.accountId",",
      " "api-key": "$context.identity.apiKey",",
      " "caller": "$context.identity.caller",",
      " "user": "$context.identity.user",",
      " "user-arn": "$context.identity.userArn",",
      " "queryString": "$input.params().querystring",",
      " "headers": "$input.params().header",",
      " "pathParams": "$input.params().path",",
      "}"
    ] ]
  }
}

cfn-api-gateway-integration-response – defines how to send the response back to the API client

ResponseParameters – a mapping from the integration response to the method response declared above. Notice the CORS headers that are necessary since the hostname for the API Gateway is different from the hostname provided in the Route53 resource record for the S3 bucket. Without these, the browser will deny AJAX request from the site to these APIs. You can read more about CORS in the API Gateway Developer Guide. Also notice that the response Content-Type is pulled from the contentType attribute in the JSON object returned from the Lambda function:
```
"ResponseParameters": {
  "method.response.header.Access-Control-Allow-Origin": "'*'",
  "method.response.header.Access-Control-Allow-Methods": "'GET, OPTIONS'",
  "method.response.header.Content-Type": "integration.response.body.contentType"
}
```
ResponseTemplates – a template of how to create the response from the Lambda invocation. In the reference application, the Lambda function returns a JSON object with a payload attribute containing a Base64 encoding of the response payload:
```
"ResponseTemplates": {
  "application/json": "$util.base64Decode( $input.path('$.payload') )"
}
```

Additionally, there are two deployment resources created, one for test and one for production. Here is an example of one:

cfn-api-gateway-deployment – a deployment has a name that will be the prefix for all resources defined.

StageName – “test” or “prod”
Variables – a list of variables for the stage. This is where the function name and version are defined for the integration URI defined above:
```
"Variables": {
  "AppFunctionName": "MyFunctionName”,
  "AppVersion": "prod"
}
```

Deploy via Swagger

I would prefer to replace most of the above API Gateway CloudFormation with a Swagger file in the application source repository and have the pipeline use the import tool to create API Gateway deployments from the Swagger. There are a few challenges with this approach. First, the creation of the Swagger requires including the AWS extensions which has a bit of a learning curve. This challenge is made easier by the fact that you can create the API Gateway via the console and then export Swagger. The other challenge is that the import tool is a Java based application that requires Maven to run. This may be difficult to get working in a Lambda invocation from CodePipeline especially given the 300 second timeout. I will however be spending some time researching this option and will blog about the results.

Stay Tuned!

Now that we have all the resources in place to deploy our application to, we can build a serverless continuous delivery pipeline with CodePipeline and Lambda. Next week we conclude this series with the third and final part looking at the details of the CloudFormation template for the pipeline and each stage of the pipeline as well as the Lambda functions that support them. Be sure to check it out!

Resources

Stelligent Amazon Pollycast