Kes: Making deployment with CloudFormation Fun

Kes helps with managing and deploying AWS resources using CloudFormation.

It makes it much easier to deploy lambda functions and create API gateway resources.

Installation

  $ npm install -g kes
  $ kes -h

   Usage: kes TYPE COMMAND [options]

   Start a Kes project

 Options:

 -V, --version                 output the version number
 -p, --profile <profile>       AWS profile name to use for authentication
 --role <role>                 AWS role arn to be assumed for the deployment
 -c, --config <config>         Path to config file
 --env-file <envFile>          Path to env file
 --cf-file <cfFile>            Path to CloudFormation templateUrl
 -t, --template <template>      A kes application template used as the base for the configuration
 --kes-class <kesClass>        Kes Class override
 -k, --kes-folder <kesFolder>  Path to config folder
 -r, --region <region>         AWS region
 --stack <stack>               stack name, defaults to the config value
 --showOutputs                  Show the list of a CloudFormation template outputs
 --yes                          Skip all confirmation prompts
 -d, --deployment <deployment>  Deployment name, default to default
 --noRollback                 Don't delete Cloudformation stacks that fail to deploy
 -h, --help                    output usage information

 Commands:
 cf [deploy|validate|compile]  CloudFormation Operations:
   create    Creates the CF stack (deprecated, start using deploy)
   update    Updates the CF stack (deprecated, start using deploy)
   upsert    Creates the CF stack and Update if already exists (deprecated, start using deploy)
   deploy    Creates the CF stack and Update if already exists
   delete    Delete the CF stack
   validate  Validates the CF stack
   compile   Compiles the CF stack
   lambda <lambdaName>                         uploads a given lambda function to Lambda service

Setting Up the First Project

Go to your project directory and run the following command.

$ npm init

This will create a .kes folder on your project folder. It will include the following files:

The cloudformation.template.yml and config.yml are required files. The variables in config.yml are parsed and used to generate the cloudformation.yml. By default, the default section of the config.yml is parsed and used in cloudformation.template.yml. If another deployment is specified in the config.yml the values of that deployment overrides the values of default file which is sent to AWS CloudFormation to create and update the stack.

CF Stack Name

The Cloudformation stack name is the same as stackName in config.yml.

Parameters

To pass parameters to the CloudFormation template, use the parameters key in config.yml. Example:

# config.yml
default:
  stackName: myStack
  parameters:
    - name: MyParameter
      value: someValue

# cloudformation.template.yml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'stack: {{stackName}} | deployed by Kes'
Parameters:
  MyParameter:
    Type: String
    Description: 'My parameter'

CF Capabailities

To pass capabilities such as CAPABILITY_IAM use capabilities key:

# config.yml
default:
  stackName: myStack
  parameters:
    - name: MyParameter
      value: someValue
  capabilities:
    - CAPABILITY_IAM

CloudFormation Tagging

To manage tags associated with your CloudFormation stack, use the tags key:

# config.yml
default:
  tags:
    color: orange
    tree: oak

Lambda Functions

To add lambda functions, use lambdas key and add them as array object. The lambda function code can be either a folder or file on your computer or a zip file on aws.

Note: In version 2.0.0 of kes, the lambda packaging is handled natively in nodejs. If you point the source to a directory, the directory is saved at the root of the zip package. This changes how handler path should be setup.

For example, if the index.js is located at /path/to/package/index.js and source: /path/to/package/index.js, the handler should be handler: index.handler.

Required Fields:

• name
• handler
• source/s3Source

Env Variables: You can add env variables to each lambda function as shown in the example below.

Example:

# config.yml
default:
  stackName: myStack
  parameters:
    - name: MyParameter
      value: someValue
  capabilities:
    - CAPABILITY_IAM

  lambdas:
    - name: myLambda1
      handler: index.handler
      timeout: 200
      source: 'node_modules/someNpmPackage'
    - name: myLambda2
      handler: package.handler
      timeout:100
      s3Source:
        bucket: mybucket
        key: mylambda.zip
      envs:
        DEBUG: true

Note:

Adding lambda functions in the config.yml has no effect unless you add the relevant CF syntax to cloudformation.template.yml

Handlebar Helpers

We use Handlebar for templating a CF template.

Each

# config.yml
default:
  myArray:
    - name: name1
    - name: name2

# cloudformation.template.yml
Resources:

{{# each myArray}}
  {{name}}:
    Type: SomeAWSResource
{{/each}}

If/else

# config.yml
default:
  myArray:
    - name: name1
      runtime: python2.7
    - name: name2

# cloudformation.template.yml
Resources:

{{# each myArray}}
  {{name}}:
    Type: SomeAWSResource
    Properties:
      Runtime: {{# if runtime}}{{runtime}}{{else}}nodejs6.10{{/if}}
{{/each}}

Each for Objects

# config.yml
default:
  myArray:
    - DEBUG: true

# cloudformation.template.yml
Resources:

{{# each myArray}}
  Lambda:
    Type: SomeAWSResource
    Properties:
      Environments:
        - {{@key}}: {{this}}
{{/each}}

Deployment

To create a CF stack or update and existing one run

 kes cf deploy

Delete an existing stack

To delete an existing stack:

  kes cf delete

Different deployment configurations

You can configure different values for different deployments. For example you might want to configure your test deployment differently from your staging and production deployments. Here is how to achieve it:

# config.yml
default:
  stackName: myStack-test
  myArray:
    - DEBUG: true

staging:
  stackName: myStack-staging
  myArray:
    - DEBUG: false

To deploy a stack with the staging configuration run:

kes cf deploy --deployment staging

Deployment Using IAM Role

You can specify an IAM role for the deployment using --role option or by setting AWS_DEPLOYMENT_ROLE environment variable.

Note: You still need an aws user with AssumeRole permission for this to work

kes cf deploy --profile myUser --role arn:aws:iam::00000000000:role/myDeploymentRole

Updating One Lambda Function

To update one lambda function outside of CF

 kes lambda myLambda

Use Templates

Kes enables you to distribute your AWS applications built with kes using a concept called template. A template is essentially a .kes folder with a cloudformation.template.yml, a config.yml and a kes.js if needed.

The user of a template can point to your template folder with the --template flag and the kes command will use the template to build the cloudformation.yml.

The user still has the option of creating her own config.yml and cloudformation.template.yml. Any variables in these files will override existing ones in the template or append it if it doesn't exist.

This setup gives users of the templates a great degree of flexibility and ownership.

Nested Templates

Kes supports use of Cloudformation Nested Templates. To use nested templates, create a separate template.yml and config.yml files for each nested template using the same rules explained above. Then include references in your main config.yml.

All nested templates will receive the parent configuration under the parent key.

Example

# config.yml
default:
  stackName: myStack-test
  myArray:
    - DEBUG: true
  nested_templates:
    myNestedTemplate:
      cfFile: /path/to/myNestedTemplate.template.yml
      configFile: /path/to/myNestedConfig.yml

staging:
  stackName: myStack-staging
  myArray:
    - DEBUG: false

# myNestedConfig.yml
default:
  timeout: 300

# myNestedTemplate.template.yml
Resources:

{{# each parent.myArray}}
  Lambda:
    Type: SomeAWSResource
    Properties:
      Timeout: {{../timeout}}
      Environments:
        - {{@key}}: {{this}}
{{/each}}

Builds templates nested in the main template using the specified config and cf file paths

Builds, uploads and deploy a Cloudformation based on options passed from the commander library

Builds and uploads a lambda function based on the options passed by the commander

The main Kes class. This class is used in the command module to create the CLI interface for kes. This class can be extended in order to override and modify the behavior of kes cli.

This class handles reading and parsing configuration files. It primarily reads config.yml and .env files

Zips a list of files or directories

Executes shell commands synchronously and logs the stdout to console.

Updates region of an AWS configuration and point to the correct of profile on ~/.aws/credentials file if necessary

Checks if the input is a file, if it is a file, it reads it and return the content, otherwise just pass the input as an output

Merges two yaml files. The merge is done using lodash.merge and it happens recursively. Meaning that values of file2 will replace values of file 1 if they have the same key.

Attempt to load a Kes override class.

Based on the information passed from the CLI by the commander module this function determines whether to use the default Kes class or use the override class provided by the user

In case of error logs the error and exit with error 1

Exists the process when called

Discover and returns the system bucket used for deployment

Copy, zip and upload lambda functions to S3

A simple helper for running a function if local is passed as argument