Wrangling CloudFormation with Sceptre

Wrangling CloudFormation with Sceptre

Anyone who has delved into CloudFormation knows its power for describing and managing your cloud infrastructure within AWS. Likewise, if you’ve spent any time writing CloudFormation templates of any significance you’ll know that you’ll spend a lot of time duplicating sections of templates.

We always aim to reduce repetition in code so this can be a bit grating.

In this post, I hope to explore a few technologies that can help with this, primarily a tool called Sceptre from Cloudreach.

Tooling

As cloud technologies have risen to prominence in the last few years, the tooling to manage them has been evolving at a similar rapid cadence.

Opinionated tools like Terraform that manage both the configuration and state of your environment are becoming particularly popular. These tools are not without issue though.

If you happen to lose your state files or people make manual changes outside of them, it can lead to serious issues with getting back on track. Another issue is that these tools often lag behind cloud platform providers in their feature set.

I recently had to write some CloudFormation templates from scratch and thought it would be a good time to take a look at what is out there in terms of tools that will allow me to quickly scaffold a project and cover some of the gaps in native CloudFormation.

I found a tool called Sceptre from Cloudreach. Sceptre is a tool that allows you to manage CloudFormation stacks through a number of templating technologies. It provides a consistent way to provide inputs to these templates. On top of this, there are a number of nice helper utilities that extend CloudFormation.

Let’s take a look through a simple demo.

Project architecture

At the base of the Sceptre project, you are going to have two main directories: they are config and templates.

The templates directory is where your CloudFormation templates are going to live, pretty straightforward.

In the config directory, you put all your YAML files that provide input to your templates. Sceptre allows you to nest multiple folders within config to separate configuration for different environments (prod, dev etc..). These are referred to as stack configs.

In this post, I’m going to look at a simple environment where we create a VPC and some security groups.

directorytree

The first thing that’s going to stand out here for someone not familiar with Sceptre is there’s a .py file within the templates folder. This is no mistake. Sceptre allows you to use native CloudFormation templates (JSON/YAML), Jinja2 templates and also Python (particularly with CloudFormation DSLs like Troposphere).

Where possible, my preference is to use native CloudFormation. I know I won’t run into an issue where my favourite DSL of the moment is lagging behind AWS in terms of its feature set. Having said that, sometimes it’s nice to use something like Jinja2 or Troposphere when you need to tap into a bit of added complexity to avoid manually copying and pasting 100 extra lines of JSON/YAML into your template.

Basic configuration

For starters we will have a base config file, I’m going to choose to put this in config/prod. For this demo, “prod” will be our environment. I’ll start with config.yaml. This file specifies a number of important items for our project:

sceptredemoprofile

project_code is the name of our project: this will be prepended to our CloudFormation stacks.

profile refers to an AWS profile within ~/.aws/config (this will be the profile that gives you access to create resources in your AWS account).

region is the AWS region you want to create your resources in.

VPC configuration

Next up is our VPC.

Under config/prod, we have vpc.yaml

vpcconfig

template_path specifies the path of the template file we are going to use these configuration values in.

Under parameters we have a number of values. These values are just passed as standard CloudFormation parameters. Take the CidrBlock parameter for example.

In my vpc.yaml template I have:

cidrblock

Then for my VPC resource:

vpctemplate

To create our stack we run (from the root of our project directory):

sceptre create-stack prod VPC

In our terminal

vpccreatecomplete

In our AWS console

sceptrevpccomplete

Pretty standard stuff up to this point, but the project scaffolding is nice out of the box to get us up and running quickly.

Security Group configuration

So for this example, I want a security group that’s going to be attached to a load balancer. This group is going to allow HTTPS for CIDR blocks that are peered to our account. This list of CIDRs could change regularly. Do we really want to maintain a list of CIDRs that can connect to our ELB by duplicating the same lines over and over in a normal CloudFormation template? Nah… in this case Sceptre allows us to pull in the complexity as we need it. For this, I’ll be going for a Python template using Troposphere.

Let’s start with our configuration file, security-groups.yaml

In this case we can’t use parameters to pass values to a Python template. Instead, Sceptre provides ‘sceptre_user_data’. We can nest parameters under this, then reference them in our templates.

sgconfigyaml

Under template_path we specify our Python file.

We can see a new section, dependencies. This tells Sceptre that the referenced stack must be created before creating this template.

Under the sceptre_user_data field, we see vpc_id. Sceptre uses the concept of resolvers to pass data into our configuration files. !stack_output allows us to pull in data from another stack we have created, in the case the prod/vpc stack.

Next, we have groups. Under here you can see two security groups that we are going to create, app_sg and elb_sg. Of particular interest is the ingress_rules section of the elb_sg. We are going to specify multiple CIDRs that we want to have inbound HTTPS access to our load balancer.

With standard CloudFormation, we would have to repeat this rule over and over in our template with slight modifications to each rule. Here I’m going to turn to Troposphere inside a Python template.

Python template

Setup

Firstly we need to add a sceptre_handler function to our Python template. This will return our generated template to Sceptre as a string which it will then use to create our CloudFormation stack.

sceptrehandler

Here are my imports

tropoimports

Stack Class

I won’t go through the whole class here. You can find a link to the code here

pythontemplate

The main logic for our template lives in the create_security_groups method. The groups variable will contain all nested groups in our YAML config file. We loop through each one using t.add_resource to add a CloudFormation “SecurityGroup” to our template. Below this, we look to see if the group has any ingress or egress rules defined. If it does, then we will loop through each one and add it to our security group.

The beauty of this is we no longer have to touch our template. We can add security groups or ingress/egress simply by modifying our YAML configuration file.

Creating security groups

To create our stack we will run “sceptre create-stack prod security-groups”

sgcreaterules

As we can see, we have a new security group “elbsg” and six new rules added. Let’s add another CIDR and update our stack.

cidr7

Now we’ll run “sceptre update-stack prod security-groups”

sg7rule

Refreshing our security group in the console, we see our new rule.

How about a new security group? Say we want a group we will attach to a lambda that lets us send requests to our servers in our web application group.

We’ll add another security group to our config file

lambdasgconfig

Then run a stack update

lambdasgconsole

And we have a new security group without touching our template.

Wrapping up

If you’re working with infrastructure in AWS then CloudFormation is an invaluable tool for managing your environment, and with almost weekly updates AWS are constantly improving on it. It’s not without its pain points, though. I hope I’ve been able to provide some insight into how tools like Sceptre and Troposphere can aid in filling in some of these gaps.

Below I have added some links that cover some of the more advanced features of Sceptre.

Links

https://sceptre.cloudreach.com/latest/docs/get_started.html

https://sceptre.cloudreach.com/latest/docs/hooks.html

https://sceptre.cloudreach.com/latest/docs/resolvers.html

Build Better Infrastructure with Sceptre and CloudFormation on AWS

https://github.com/densikat-shinesolutions/sceptre-example

 

david.ensikat@shinesolutions.com

DevOps Engineer at Shine Solutions

No Comments

Leave a Reply