Kes: Making deployment with CloudFormation Fun

Kes helps with managing and deploying AWS resources using CloudFormation.

It makes it much easier to deploy lambda functions and create API gateway resources.

Installation

  $ npm install -g kes
  $ kes -h

   Usage: kes TYPE COMMAND [options]

   Start a Kes project

 Options:

 -V, --version                 output the version number
 -p, --profile <profile>       AWS profile name to use for authentication
 --role <role>                 AWS role arn to be assumed for the deployment
 -c, --config <config>         Path to config file
 --env-file <envFile>          Path to env file
 --cf-file <cfFile>            Path to CloudFormation templateUrl
 -t, --template <template>      A kes application template used as the base for the configuration
 --kes-class <kesClass>        Kes Class override
 -k, --kes-folder <kesFolder>  Path to config folder
 -r, --region <region>         AWS region
 --stack <stack>               stack name, defaults to the config value
 --showOutputs                  Show the list of a CloudFormation template outputs
 --yes                          Skip all confirmation prompts
 -d, --deployment <deployment>  Deployment name, default to default
 -h, --help                    output usage information

 Commands:
 cf [deploy|validate|compile]  CloudFormation Operations:
   create    Creates the CF stack (deprecated, start using deploy)
   update    Updates the CF stack (deprecated, start using deploy)
   upsert    Creates the CF stack and Update if already exists (deprecated, start using deploy)
   deploy    Creates the CF stack and Update if already exists
   delete    Delete the CF stack
   validate  Validates the CF stack
   compile   Compiles the CF stack
   lambda <lambdaName>                         uploads a given lambda function to Lambda service

Setting Up the First Project

Go to your project directory and run the following command.

$ npm init

This will create a .kes folder on your project folder. It will include the following files:

file description
.env This optional file can hold your project secrets and should not be committed
cloudformation.template.yml A base CF template written with Mustache/Handlebar templating language
config.yml The main required configuration file for a kes deployment
kes.js An optional Kes class override that can change how Kes class is used

The cloudformation.template.yml and config.yml are required files. The variables in config.yml are parsed and used to generate the cloudformation.yml. By default, the default section of the config.yml is parsed and used in cloudformation.template.yml. If another deployment is specified in the config.yml the values of that deployment overrides the values of default file which is sent to AWS CloudFormation to create and update the stack.

CF Stack Name

The Cloudformation stack name is the same as stackName in config.yml.

Parameters

To pass parameters to the CloudFormation template, use the parameters key in config.yml. Example:

# config.yml
default:
  stackName: myStack
  parameters:
    - name: MyParameter
      value: someValue
# cloudformation.template.yml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'stack: {{stackName}} | deployed by Kes'
Parameters:
  MyParameter:
    Type: String
    Description: 'My parameter'

CF Capabailities

To pass capabilities such as CAPABILITY_IAM use capabilities key:

# config.yml
default:
  stackName: myStack
  parameters:
    - name: MyParameter
      value: someValue
  capabilities:
    - CAPABILITY_IAM

CloudFormation Tagging

To manage tags associated with your CloudFormation stack, use the tags key:

# config.yml
default:
  tags:
    color: orange
    tree: oak

Lambda Functions

To add lambda functions, use lambdas key and add them as array object. The lambda function code can be either a folder or file on your computer or a zip file on aws.

Note: In version 2.0.0 of kes, the lambda packaging is handled natively in nodejs. If you point the source to a directory, the directory is saved at the root of the zip package. This changes how handler path should be setup.

For example, if the index.js is located at /path/to/package/index.js and source: /path/to/package/index.js, the handler should be handler: index.handler.

Required Fields:

  • name
  • handler
  • source/s3Source

Env Variables: You can add env variables to each lambda function as shown in the example below.

Example:

# config.yml
default:
  stackName: myStack
  parameters:
    - name: MyParameter
      value: someValue
  capabilities:
    - CAPABILITY_IAM

  lambdas:
    - name: myLambda1
      handler: index.handler
      timeout: 200
      source: 'node_modules/someNpmPackage'
    - name: myLambda2
      handler: package.handler
      timeout:100
      s3Source:
        bucket: mybucket
        key: mylambda.zip
      envs:
        DEBUG: true

Note:

Adding lambda functions in the config.yml has no effect unless you add the relevant CF syntax to cloudformation.template.yml

Handlebar Helpers

We use Handlebar for templating a CF template.

Each

# config.yml
default:
  myArray:
    - name: name1
    - name: name2
# cloudformation.template.yml
Resources:

{{# each myArray}}
  {{name}}:
    Type: SomeAWSResource
{{/each}}

If/else

# config.yml
default:
  myArray:
    - name: name1
      runtime: python2.7
    - name: name2
# cloudformation.template.yml
Resources:

{{# each myArray}}
  {{name}}:
    Type: SomeAWSResource
    Properties:
      Runtime: {{# if runtime}}{{runtime}}{{else}}nodejs6.10{{/if}}
{{/each}}

Each for Objects

# config.yml
default:
  myArray:
    - DEBUG: true
# cloudformation.template.yml
Resources:

{{# each myArray}}
  Lambda:
    Type: SomeAWSResource
    Properties:
      Environments:
        - {{@key}}: {{this}}
{{/each}}

Deployment

To create a CF stack or update and existing one run

 kes cf deploy

Delete an existing stack

To delete an existing stack:

  kes cf delete

Different deployment configurations

You can configure different values for different deployments. For example you might want to configure your test deployment differently from your staging and production deployments. Here is how to achieve it:

# config.yml
default:
  stackName: myStack-test
  myArray:
    - DEBUG: true

staging:
  stackName: myStack-staging
  myArray:
    - DEBUG: false

To deploy a stack with the staging configuration run:

kes cf deploy --deployment staging

Deployment Using IAM Role

You can specify an IAM role for the deployment using --role option or by setting AWS_DEPLOYMENT_ROLE environment variable.

Note: You still need an aws user with AssumeRole permission for this to work

kes cf deploy --profile myUser --role arn:aws:iam::00000000000:role/myDeploymentRole

Updating One Lambda Function

To update one lambda function outside of CF

 kes lambda myLambda

Use Templates

Kes enables you to distribute your AWS applications built with kes using a concept called template. A template is essentially a .kes folder with a cloudformation.template.yml, a config.yml and a kes.js if needed.

The user of a template can point to your template folder with the --template flag and the kes command will use the template to build the cloudformation.yml.

The user still has the option of creating her own config.yml and cloudformation.template.yml. Any variables in these files will override existing ones in the template or append it if it doesn't exist.

This setup gives users of the templates a great degree of flexibility and ownership.

Nested Templates

Kes supports use of Cloudformation Nested Templates. To use nested templates, create a separate template.yml and config.yml files for each nested template using the same rules explained above. Then include references in your main config.yml.

All nested templates will receive the parent configuration under the parent key.

Example

# config.yml
default:
  stackName: myStack-test
  myArray:
    - DEBUG: true
  nested_templates:
    myNestedTemplate:
      cfFile: /path/to/myNestedTemplate.template.yml
      configFile: /path/to/myNestedConfig.yml

staging:
  stackName: myStack-staging
  myArray:
    - DEBUG: false
# myNestedConfig.yml
default:
  timeout: 300
# myNestedTemplate.template.yml
Resources:

{{# each parent.myArray}}
  Lambda:
    Type: SomeAWSResource
    Properties:
      Timeout: {{../timeout}}
      Environments:
        - {{@key}}: {{this}}
{{/each}}

buildNestedCfs

Builds templates nested in the main template using the specified config and cf file paths

buildNestedCfs(config: object, KesClass: object, options: object): Promise
Parameters
config (object) Kes config object
KesClass (object) the Kes Class
options (object) The options passed by the commander library
Returns
Promise: returns a promise of an updated Kes config object

buildCf

Builds, uploads and deploy a Cloudformation based on options passed from the commander library

buildCf(options: object, cmd: string): undefined
Parameters
options (object) Options passed by the commander library
cmd (string) the argument selected in the CLI, e.g. deploy, update, etc.
Returns
undefined:

buildLambda

Builds and uploads a lambda function based on the options passed by the commander

buildLambda(options: object, cmd: string): undefined
Parameters
options (object) Options passed by the commander library
cmd (string) the argument selected in the CLI, e.g. lambda name
Returns
undefined:

Kes

The main Kes class. This class is used in the command module to create the CLI interface for kes. This class can be extended in order to override and modify the behavior of kes cli.

new Kes(config: Object)
Parameters
config (Object) an instance of the Config class (config.js)
Example
const { Kes, Config } = require('kes');

const options = { stack: 'myStack' };
const config = new Config(options);
const kes = new Kes(config);

// create a new stack
kes.deployStack()
 .then(() => describeCF())
 .then(() => updateSingleLambda('myLambda'))
 .catch(e => console.log(e));
Instance Members
describeStack(stackName)
updateSingleLambda(name)
compileCF()
uploadToS3(bucket, key, body)
uploadCF()
waitFor(wait)
cloudFormation()
validateTemplate()
describeCF()
deleteCF()
opsStack()
upsertStack()
deployStack()
createStack()
updateStack()
deleteStack()

Config

This class handles reading and parsing configuration files. It primarily reads config.yml and .env files

new Config(options: Object, stack: String, deployment: String, configFile: String, envFile: String)
Parameters
options (Object) a js object that includes required options.
Name Description
options.stack String? the stack name
options.deployment String (default null) the deployment name
options.region String (default 'us-east-1') the aws region
options.profile String (default null) the profile name
options.kesFolder String (default '.kes') the path to the kes folder
options.configFile String (default 'config.yml') the path to the config.yml
options.envFile String (default '.env') the path to the .env file
options.cfFile String (default 'cloudformation.template.yml') the path to the CF template
stack (String) Stack name
deployment (String) Deployment name
configFile (String) path to the config.yml file
envFile (String) path to the .env file (optional)
Example
const config = new Config('mystack', 'dev', '.kes/config.yml', '.kes/.env');
Instance Members
parse()
flatten()

zip

Zips a list of files or directories

zip(zipFile: string, srcList: array, dstPath: type): Promise
Parameters
zipFile (string) filename and path where the zip file is stored
srcList (array) array of files and directories paths
dstPath (type) for directories whether to put the directories at root of the zip file or relative to your path on the local machine
Returns
Promise:

exec

Executes shell commands synchronously and logs the stdout to console.

exec(cmd: String, verbose: Boolean): Buffer
Parameters
cmd (String) Bash command
verbose (Boolean = true) whether to post stdout to console
Returns
Buffer: The command's stdout in for of Buffer

configureAws

Updates region of an AWS configuration and point to the correct of profile on ~/.aws/credentials file if necessary

configureAws(region: String, profile: String, role: any)
Parameters
region (String = 'us-east-1') AWS region
profile (String = null) aws credentials profile name
role (any)

fileToString

Checks if the input is a file, if it is a file, it reads it and return the content, otherwise just pass the input as an output

fileToString(file: String): String
Parameters
file (String) A file path or a string
Returns
String: String content of a given file

mergeYamls

Merges two yaml files. The merge is done using lodash.merge and it happens recursively. Meaning that values of file2 will replace values of file 1 if they have the same key.

mergeYamls(file1: String, file2: String): String
Parameters
file1 (String) Yaml path to file 1 or file 1 string
file2 (String) Yaml path to file 2 or file 2 string
Returns
String: Merged Yaml file in string format

determineKesClass

Based on the information passed from the CLI by the commander module this function determines whether to use the default Kes class or use the override class provided by the user

determineKesClass(options: object, Kes: Class): Class
Parameters
options (object) The options passed by the commander library
Kes (Class) the default kes class
Returns
Class: Kes class

failure

In case of error logs the error and exit with error 1

failure(e: Error)
Parameters
e (Error) error object

success

Exists the process when called

success()

getSystemBucket

Discover and returns the system bucket used for deployment

getSystemBucket(config: Object): string
Parameters
config (Object) cumulus config object
Returns
string: name of the bucket

Lambda

Copy, zip and upload lambda functions to S3

new Lambda(config: Object, kesFolder: String, bucket: String, key: String)
Parameters
config (Object) the configuration object
kesFolder (String) the path to the .kes folder
bucket (String) the S3 bucket name
key (String) the main folder to store the data in the bucket (stack)
Instance Members
buildS3Path(lambda)
getHash(folderName, method)
zipLambda(lambda)
uploadLambda(lambda)
zipAndUploadLambda(lambda)
process()
updateSingleLambda(name)

localRun

A simple helper for running a function if local is passed as argument

localRun(func: Function): Object
Parameters
func (Function) A javascript function
Returns
Object: returns the result of the function call
Example
// test.js
const { localRun } = require('kes');
localRun(() => {
  console.log('my function');
});
// result
// $ node test.js local
// my function