29 Aug 2018 Wrangling CloudFormation with Sceptre
Anyone who has delved into CloudFormation knows its power for describing and managing your cloud infrastructure within AWS. Likewise, if you’ve spent any time writing CloudFormation templates of any significance you’ll know that you’ll spend a lot of time duplicating sections of templates.
We always aim to reduce repetition in code so this can be a bit grating.
In this post, I hope to explore a few technologies that can help with this, primarily a tool called Sceptre from Cloudreach.
Tooling
As cloud technologies have risen to prominence in the last few years, the tooling to manage them has been evolving at a similar rapid cadence.
Opinionated tools like Terraform that manage both the configuration and state of your environment are becoming particularly popular. These tools are not without issue though.
If you happen to lose your state files or people make manual changes outside of them, it can lead to serious issues with getting back on track. Another issue is that these tools often lag behind cloud platform providers in their feature set.
I recently had to write some CloudFormation templates from scratch and thought it would be a good time to take a look at what is out there in terms of tools that will allow me to quickly scaffold a project and cover some of the gaps in native CloudFormation.
I found a tool called Sceptre from Cloudreach. Sceptre is a tool that allows you to manage CloudFormation stacks through a number of templating technologies. It provides a consistent way to provide inputs to these templates. On top of this, there are a number of nice helper utilities that extend CloudFormation.
Let’s take a look through a simple demo.
Project architecture
At the base of the Sceptre project, you are going to have two main directories: they are config and templates.
The templates directory is where your CloudFormation templates are going to live, pretty straightforward.
In the config directory, you put all your YAML files that provide input to your templates. Sceptre allows you to nest multiple folders within config to separate configuration for different environments (prod, dev etc..). These are referred to as stack configs.
In this post, I’m going to look at a simple environment where we create a VPC and some security groups.
The first thing that’s going to stand out here for someone not familiar with Sceptre is there’s a .py file within the templates folder. This is no mistake. Sceptre allows you to use native CloudFormation templates (JSON/YAML), Jinja2 templates and also Python (particularly with CloudFormation DSLs like Troposphere).
Where possible, my preference is to use native CloudFormation. I know I won’t run into an issue where my favourite DSL of the moment is lagging behind AWS in terms of its feature set. Having said that, sometimes it’s nice to use something like Jinja2 or Troposphere when you need to tap into a bit of added complexity to avoid manually copying and pasting 100 extra lines of JSON/YAML into your template.
Basic configuration
For starters we will have a base config file, I’m going to choose to put this in config/prod. For this demo, “prod” will be our environment. I’ll start with config.yaml. This file specifies a number of important items for our project:
project_code is the name of our project: this will be prepended to our CloudFormation stacks.
profile refers to an AWS profile within ~/.aws/config (this will be the profile that gives you access to create resources in your AWS account).
region is the AWS region you want to create your resources in.
VPC configuration
Next up is our VPC.
Under config/prod, we have vpc.yaml
template_path specifies the path of the template file we are going to use these configuration values in.
Under parameters we have a number of values. These values are just passed as standard CloudFormation parameters. Take the CidrBlock parameter for example.
In my vpc.yaml template I have:
Then for my VPC resource:
To create our stack we run (from the root of our project directory):
sceptre create-stack prod VPC
In our terminal
In our AWS console
Pretty standard stuff up to this point, but the project scaffolding is nice out of the box to get us up and running quickly.
Security Group configuration
So for this example, I want a security group that’s going to be attached to a load balancer. This group is going to allow HTTPS for CIDR blocks that are peered to our account. This list of CIDRs could change regularly. Do we really want to maintain a list of CIDRs that can connect to our ELB by duplicating the same lines over and over in a normal CloudFormation template? Nah… in this case Sceptre allows us to pull in the complexity as we need it. For this, I’ll be going for a Python template using Troposphere.
Let’s start with our configuration file, security-groups.yaml
In this case we can’t use parameters to pass values to a Python template. Instead, Sceptre provides ‘sceptre_user_data’. We can nest parameters under this, then reference them in our templates.
Under template_path we specify our Python file.
We can see a new section, dependencies. This tells Sceptre that the referenced stack must be created before creating this template.
Under the sceptre_user_data field, we see vpc_id. Sceptre uses the concept of resolvers to pass data into our configuration files. !stack_output allows us to pull in data from another stack we have created, in the case the prod/vpc stack.
Next, we have groups. Under here you can see two security groups that we are going to create, app_sg and elb_sg. Of particular interest is the ingress_rules section of the elb_sg. We are going to specify multiple CIDRs that we want to have inbound HTTPS access to our load balancer.
With standard CloudFormation, we would have to repeat this rule over and over in our template with slight modifications to each rule. Here I’m going to turn to Troposphere inside a Python template.
Python template
Setup
Firstly we need to add a sceptre_handler function to our Python template. This will return our generated template to Sceptre as a string which it will then use to create our CloudFormation stack.
Here are my imports
Stack Class
I won’t go through the whole class here. You can find a link to the code here
The main logic for our template lives in the create_security_groups method. The groups variable will contain all nested groups in our YAML config file. We loop through each one using t.add_resource to add a CloudFormation “SecurityGroup” to our template. Below this, we look to see if the group has any ingress or egress rules defined. If it does, then we will loop through each one and add it to our security group.
The beauty of this is we no longer have to touch our template. We can add security groups or ingress/egress simply by modifying our YAML configuration file.
Creating security groups
To create our stack we will run “sceptre create-stack prod security-groups”
As we can see, we have a new security group “elbsg” and six new rules added. Let’s add another CIDR and update our stack.
Now we’ll run “sceptre update-stack prod security-groups”
Refreshing our security group in the console, we see our new rule.
How about a new security group? Say we want a group we will attach to a lambda that lets us send requests to our servers in our web application group.
We’ll add another security group to our config file
Then run a stack update
And we have a new security group without touching our template.
Wrapping up
If you’re working with infrastructure in AWS then CloudFormation is an invaluable tool for managing your environment, and with almost weekly updates AWS are constantly improving on it. It’s not without its pain points, though. I hope I’ve been able to provide some insight into how tools like Sceptre and Troposphere can aid in filling in some of these gaps.
Below I have added some links that cover some of the more advanced features of Sceptre.
Links
https://sceptre.cloudreach.com/latest/docs/get_started.html
https://sceptre.cloudreach.com/latest/docs/hooks.html
https://sceptre.cloudreach.com/latest/docs/resolvers.html
Build Better Infrastructure with Sceptre and CloudFormation on AWS
https://github.com/densikat-shinesolutions/sceptre-example
No Comments