Dancing the Lambada with AWS Lambda or sending emails on S3 events

Introduction

The aim of this post is to offer a quick start to using AWS Lambda by using a small application for illustration. It is meant for someone who has some experience using Node.js (although I had very little when I started) and Javascript. So please forgive any non-JS idioms.

AWS Lambda is presented by Amazon as a service to run short-lived Node.js processes that are triggered in response to events that occur in your Amazon infrastructure. Of course you can do this already using home grown methods. But by using Lambda you do not have to have an EC2 instance running or support tasks such as scaling Node.js and backups.

As with any service there are limitations that you should be aware of. The most important are:
1. You have no direct access to the infrastructure
2. Only Node.js (Javascript) applications are supported. There are hints this may change since the descriptor for an application has a field for the runtime. Ex. “Runtime”: “nodejs”
3. Applications can only run for a maximum of 60 secs before they are killed.

For those who just want to get to the code, here it is: https://github.com/rmauge/aws-lambda-s3-email

Setup

The project README.md has all the tasks to install a complete development environment. This post will reference it as necessary so consider the README.md the source. You need to have an AWS http://aws.amazon.com/ account to try this out. I use Ubuntu so the commands expect that environment but it should be fairly straight forward to adapt for other OSes. Complete up to and including step 4 before continuing.

1. Install pip
* Download: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
* Install: sudo python get-pip.py
2. Use a python virtual environment
* Install: sudo pip install virtualenv
* Create: virtualenv lambda-env
* Activate: source lambda-env/bin/activate
3. Install node using a virtual environment
* Install nodeenv: pip install nodeenv
* Install node: nodeenv -p –node=0.10.32 nenv
* Update npm (installed with node): npm install npm -g
4. Install Amazon AWS [cli tools](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html)
* Install: pip install awscli
* Configure: aws configure
5. Update Lambda Function
* Clone project: git@github.com:rmauge/aws-lambda-s3-email.git
* Zip directory:
“`
cd aws-lambda-s3-email/ && zip -r ../aws-lambda-s3-email.zip . -x build/ test/ *~ *.git* .gitignore && cd ..
“`
* Submit update:
“`
aws lambda upload-function
–region us-east-1
–function-name lambdaSubmissionFunction
–function-zip aws-lambda-s3-email.zip
–role arn:aws:iam::99999999:role/lamdba_exec_role
–mode event
–handler index.handler
–runtime nodejs
–debug
–timeout 10
–memory-size 128
“`
6. Test function manually
* “`
aws lambda invoke-async –function-name lambdaSubmissionFunction –region us-east-1 –invoke-args aws-lambda-s3-email/test/submissionexample.txt –debug
“`
* When testing the key must exist in the S3 bucket

AWS Policies and Roles

The Lambda Function requires certain roles/policies to work. These are:
An IAM (AWS Identity and Access Management) user to send email (ses-smtp-user)
A role and policy allowing a bucket event to invoke a Lambda Function (lambda_invoke_role)
A role and policy allowing the Lambda Function to access the S3 bucket and logging (lamdba_exec_role)

In a nutshell you need to install Node.js http://nodejs.org/ and Amazon CLI tools http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html

I used a python virtual environment to isolate changes but how you install is up to you.

On the Amazon side you must have the proper users, roles and policies set up so that the various pieces can do their work. But we will get to this later on.

At step 5 you can edit the Lambda Function and settings after cloning the repo to reflect your AWS settings. The config.js file is illustrative.
Since the Lambda Function needs permissions to do it’s work, we have to setup permissions to the S3 buckets and the logs.

Upload and Configure the Lambda Function

We will create a role in IAM that has the required policy:
L1. Go to the IAM dashboard https://console.aws.amazon.com/iam/home?region=us-east-1#home
L2. Create a new role (the “exec” role) with a new policy attached or inline. This is the role that the function assumes when running. Ex policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::your-bucket/*"
]
}
]
}

You can do this by in IAM by creating a new role with a chosen name “lambda_exec_role” for example. Role Type: “AWS Lambda”. Then attaching the policies “AmazonS3FullAccess” and “CloudWatchLogsFullAccess”.

Make note of the Role ARN (Amazon Resource Name) after it is created, you will need it later.

L3. We will also need a role for the S3 bucket to assume to send an event to the function.
Create another role (the “invoke” role) with the following policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Resource": [
"*"
],
"Action": [
"lambda:InvokeFunction"
]
}
]
}

Also only allow the S3 buckets to assume this role. This is the trusted entity profile.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"ArnLike": {
"sts:ExternalId": "arn:aws:s3:::*"
}
}
}
]
}

You can do this by in IAM by creating a new role with a chosen name “lambda_invoke_role”. Role Type: “AWS S3 Invocation for Lambda Functions”. Policies: AWSLambdaRole.

Now you can zip and upload the function as in Step 5 with the role ARN, the exec_role, as recorded in L2.
--role arn:aws:iam::99999999:role/lamdba_exec_role

If the upload is successful then you need to get the ARN of the function. It should be in results of the aws lambda upload-function command just run. Or you can find it in the console https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions.
Make a note of it for later.

Configure S3 Bucket events

The S3 bucket now has to be configured to call that function with that role when an event such as a post, put, etc takes place.

S1. Go to https://console.aws.amazon.com/s3/home?region=us-east-1
S2. Click on the bucket and go to properties.
S3. Under events add a new Notification
S4. Give it a name and choose the Events you need: post, put etc.
S5. Choose Send to Lambda Function and fill in the function ARN from the return value in Step 5. Fill in the Role ARN with the “lambda_invoke_role” from L3.
Save.

Setting up the Amazon Simple Email Service (SES)

Since the Lambda function sends emails these permission also have to be in place.
Make note of your SMTP settings since these are used by the npm mailer which we are using https://github.com/andris9/Nodemailer. In particular make note of the server region which may have to be configured. But since I am using us-east-1 no changes had to be made.

Create a new user to send mail.

Go to https://console.aws.amazon.com/ses/home?region=us-east-1 and create a new user “ses-smtp-user”, for example. make note of the credentials. These are used in config.js to send emails.

"SES_STMP_USERNAME" : "YOUR_SMTP_USERNAME",
"SES_SMTP_PASSWORD" : "YOUR_SMTP_PASSWORD",

You also may need to verify an email address used for sending the email. Again set in config.js.

"defaultEmailFrom" : "you@example.com",

If you are using the example, as is, you will need to zip the package up again and upload.

Email templates

I choose to store email templates in yaml rather than json format since it supports line breaks. And is easier on the eyes. The location of these are configured in config.js. And there is a sample at email-templates/templates.yml.

Testing

All should be ready now to test the function from the command line. The sample code expects an email meta key field which is used to get the email to sent to. But your case may be different.
Change the file test/submissionexample.txt to reflect your Amazon settings. Step 5 has the command to test the function.

To check on the progress take a look at the logs at https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logs:

These should alert you to any problems.

I know it has been a lot to go through but you should now have a good grasp of working with Amazon Lambda. Have fun.

Advertisements
This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s