AWS DevOps Pro Certification Blog Post Series: AWS Lambda


This is part of the blog post series: AWS DevOps Pro Certification

Caveat emptor

Using AWS costs money, some of these services may not be part of the AWS Free Tier. You can keep costs down by tearing down anything you've created whilst learning, but it's still possible to run up a hefty bill so pay attention to the instances you setup!

I'm very lucky to be able to use my employer's AWS account. You should ask your place of work if a similar arrangement can be made as part of your study.

Velocius quam asparagi conquantur

The format of the blog posts is liable to change as I try to refine my mental model of each domain, so be sure to revisit the blog posts on a regular basis.

What?

AWS Lambda is ...

Additional resources:

Why?

Here are some ideas that a DevOps / Infra team might use cases for AWS Lambda. None of this is new or groundbreaking innovations. The only difference is that when trying to implement these in AWS Lambda we no longer need to factor new servers, billing is per second and Lambda was built to talk with other AWS services in mind.

Use cases inspired by the epagon blog post: Why DevOps Engineers Love AWS

When?

How?

We're going to use a simple example where the DevOps engineer wants to log all files being uploaded for a given S3 bucket. A lot of the hard work around this tutorial was done via Sunil Dalal's blog post, "Using Lambda as S3 events processor". Thanks, Sunil for sharing!

Pre-requisites

Create files

Copy the following snippet and call it index.js

exports.handler = async (event) => {
    var srcBucket = event.Records[0].s3.bucket.name;
    var srcKey = decodeURIComponent(event.Records[0].s3.object.key);

    console.log("bucket:", srcBucket, " file: ", srcKey);
};

Copy the following snippet and call it payload-test.json

{
  "Records":[  
    {  
      "eventVersion":"2.0",
      "eventSource":"aws:s3",
      "awsRegion":"us-west-2",
      "eventTime":"1970-01-01T00:00:00.000Z",
      "eventName":"ObjectCreated:Put",
      "userIdentity":{  
        "principalId":"AIDAJDPLRKLG7UEXAMPLE"
      },
      "requestParameters":{  
        "sourceIPAddress":"127.0.0.1"
      },
      "responseElements":{  
        "x-amz-request-id":"C3D13FE58DE4C810",
        "x-amz-id-2":"FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
      },
      "s3":{  
        "s3SchemaVersion":"1.0",
        "configurationId":"testConfigRule",
        "bucket":{  
          "name":"sourcebucket",
          "ownerIdentity":{  
            "principalId":"A3NL1KOZZKExample"
          },
          "arn":"arn:aws:s3:::sourcebucket"
        },
        "object":{  
          "key":"HappyFace.jpg",
          "size":1024,
          "eTag":"d41d8cd98f00b204e9800998ecf8427e",
          "versionId":"096fKKXTRTtl3on89fVO.nfljtsv6qko"
        }
      }
    }
  ]
}

The remainder of the session can be done via the command line:

export LAMBDA_NAME=your-lambda-function-name
export LAMBDA_ROLE=your-lambda-execution-role
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r ".Account")
export LAMBDA_ROLE_ARN=arn:aws:iam::$AWS_ACCOUNT_ID:role/$LAMBDA_ROLE
export LAMBDA_S3_BUCKET=your-s3-bucket
export LAMBDA_S3_ARN=arn:aws:s3:::$LAMBDA_S3_BUCKET

# package the lambda function
zip function.zip index.js

# create function
aws lambda create-function --function-name $LAMBDA_NAME \
  --zip-file fileb://function.zip \
  --handler index.handler \
  --runtime nodejs10.x \
  --role $LAMBDA_ROLE_ARN


# setup S3 notifications, first we'll allow the s3 bucket to invoke the lambda
aws lambda add-permission \
  --function-name $LAMBDA_NAME \
  --principal s3.amazonaws.com \
  --statement-id $LAMBDA_NAME$RANDOM \
  --action "lambda:InvokeFunction" \
  --source-arn $LAMBDA_S3_ARN \
  --source-account $AWS_ACCOUNT_ID

export LAMBDA_ARN=$(aws lambda get-function --function-name $LAMBDA_NAME  | jq -r .Configuration.FunctionArn)

cat << EOF > notification.json
{
    "LambdaFunctionConfigurations": [
      {
        "Id": "1234567890",
        "LambdaFunctionArn": "$LAMBDA_ARN",
        "Events": [ "s3:ObjectCreated:*" ]
      }
    ]
}
EOF

# next create the notification event in the bucket
# DANGER: this will overwrite any existing event notifications in your bucket
# DO NOT RUN THIS on a bucket that is important to you or work!
aws s3api put-bucket-notification-configuration \
  --bucket $LAMBDA_S3_BUCKET \
  --notification-configuration file://notification.json

# test the integration to see if the message formatting is correct. 
# this should look identical to the actual CloudWatch entry.
# n.b. base64 appears to use the same switch to decode for both BSD and GNU 
# variants
aws lambda invoke \
  --invocation-type RequestResponse \
  --function-name $LAMBDA_NAME \
  --log-type Tail \
  --payload file://payload-test.json outputfile.txt \
  | jq -r .LogResult | base64 --decode


# Finally let's test it properly
touch hello.txt
aws s3 cp hello.txt s3://$LAMBDA_S3_BUCKET/

# If you didn't install awslogs, you can still use AWS Console to view logs in 
# Cloud Watch.
awslogs get /aws/lambda/s3-blab --start='5 min'

# Teardown

aws lambda delete-function --function-name $LAMBDA_NAME

aws s3api put-bucket-notification-configuration \
  --bucket $LAMBDA_S3_BUCKET \
  --notification-configuration 'LambdaFunctionConfigurations=[]'

# you may wish to clear down the S3 bucket of your test files and the log group 
# that was created in CloudWatch.

API and CLI features and verbs

### Features

### Verbs (CRUD)

### Outliers

AWS DevOps Pro Certification Blog Post Series



Tweet