AWS CloudFormation Software Development Practices

08 Sep 2023 AWS CloudFormation Software Development Practices

Have you ever wondered, how good software development & designing AWS CloudFormation templates can belong together ?

While working with CloudFormation over the years, I have discovered that understanding how good software development practices have been implemented in CloudFormation has improved my process of designing CloudFormation templates.

In this blog post, I’ll share my experience and provide a summary of the software development practices integrated into CloudFormation, offering insights on how to apply these principles to enhance your CloudFormation template design process.

Input validation

Input validation helps the user to provide the correct input parameter values to their CloudFormation Stack. CloudFormation provides multiple options for input parameter validations, which can be applied on their own or in combination. Using input parameter validation in CloudFormation not only improves user experience. It also enhances the reliability of the stack creation. It instantly reports to the user if one of the input values is not correct.

A parameter validation must be defined as a parameter property in your CloudFormation parameter.

Parameter Constraints

You can define criteria to which the input values of the stack must adhere. This helps to ensure that only valid values are provided.

AllowedPattern

You can provide a regular expression pattern that the parameter value must match. This is useful to enforce specific formats like CIDR ranges.

PrivateSubnetCidr:
  AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$

To help the user understand the constraint it is recommended that you provide a ConstraintDescription property. The description will be offered as a prompt to the developer if the pattern is not matched.

PrivateSubnetCidr:
  AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$
  ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-28

If the user provides an invalid value he will receive the message CIDR block parameter must be in the form x.x.x.x/16-28.

AllowedValues

If you need something simpler you can specify the allowed values of a parameter.

CreatePrivateSubnets:
  AllowedValues:
    - 'true'
    - 'false'

In this example only the string values of true and false are valid parameter values.

Default

You can define a default value for properties which don’t require user input. I would recommend to define as many default values as possible in your template as it improves the user experience.

CreatePrivateSubnets:
  AllowedValues:
    - 'true'
    - 'false'
Default: 'true'

In this example, a private subnet is created unless the user provides the value false.

MinLength & MaxLength

Sometimes it is useful to define a minimum or maximum length (or both) for an input parameter. This can be very useful when you need to limit the length of an input parameter value because you are passing it to a resource property which has a max length.

PrivateSubnetName:
  MinLength: 8
  MaxLength: 32
...

In this example I’m using the min and max length to limit the amount of characters the name of the created private subnet can have.

Type Annotation

Very much like many programming languages, CloudFormation enforces strict parameter type annotation.
Besides the standard data types String and Number you can also define a list data type for both. The data type List<Number> expects a comma separated array of integers or floats e.g. "1,2,3,4" and CommaDelimitedList expects a comma separated array of strings e.g. "String1,String2,String3". Furthermore CloudFormation also supports AWS Specific data types and SSM Parameter data types.

An AWS Specific data type supports the user in providing the correct input parameter values, e.g., providing a valid EC2 Keypair name or an EC2 AMI ID. A full list of all available AWS data types is available in the documentation.

This example expects the user to provide a valid EC2 AMI ID for an EC2 Instance resource

EC2ImageId:
  Type: AWS::EC2::Image::Id
...
EC2Instance:
  Type: AWS::EC2::Instance
  Properties::
    ImageId: !Ref EC2ImageId  

SSM Parameter data types expect a valid Systems Manager parameter as input value and are resolved while CloudFormation resolves the parameter section and therefore the Parameter tab in the CloudFormation console will contain the resolved value. SSM Parameter values are resolved to either a string or a list of strings and are defined as AWS::SSM::Parameter::Value<DataType>. Datatype can be any string parameter data type supported by CloudFormation like String , CommaDelimitedList or an AWS data type.

EC2ImageId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'
...
EC2Instance:
  Type: AWS::EC2::Instance
  Properties::
    ImageId: !Ref EC2ImageId   

In this example we define a SSM Parameter type and are expecting that the resolved value will be a valid EC2 AMI ID. It also has a default defined to use an AWS managed SSM Parameter, containing the latest Amazon Linux 2 AMI ID. A full list of all supported SSM Parameter data types is available in the documentation.

Rules

You can move the parameter validation to the next level by defining rules to assert the input parameter values, and even define conditions. You can allow the use of cost-intensive instance types based on the environment or do cross-parameter validation, such as checking one parameter cannot be empty if another parameter matches a certain value.

Rules:
  SubnetsInVPC:
    Assertions:
      - Assert:
          'Fn::EachMemberEquals':
            - 'Fn::ValueOf':
                - Subnets
                - VpcId
            - Ref: VpcId
        AssertDescription: All subnets must in the VPC

In this example we are using AWS rule functions to identify if the provided Subnets are a member of the provided VPC Id. If a provided subnet is not part of the provided VPC the stack creation will fail with an error message.

Built-In Variables

Built-in variables in AWS CloudFormation are called pseudo parameters and are used the same way as regular parameters. Use the Ref method to use them in your template.

Ref: AWS::Region

In this example we are refering to the built-in variable AWS::Region which will return the region in which the CloudFormation stack is being created. A list of all available built-in variables is available in the documentation.

Functions

AWS CloudFormation allows the use of functions to process the information in your template. CloudFormation offers multiple built-in functions, so called Intrinsic Functions , but also provides the ability to add additional functions by “importing libraries”, which are called Transforms, or in short macros.

Intrinsic Functions

Intrinsic functions are built-in functions in AWS CloudFormation enhancing your CloudFormation experience.

Base64 encoding

AWS CloudFormation allows you to encode strings to Base64,

Fn::Base64: String value to encode

Or encode a multiple line string:

Fn::Base64: |
  Multi line string to encode

You can even use it in combination with other functions, as long as they are returning a string. 

Fn::Base64: !Sub |
  This is my stack ${AWS::StackName} in Base64

I have never seen it used anywhere other than in the EC2 UserData, but you could store a base64 encoded string in AWS SSM Parameter Store or use AWS::CloudFormation::Init to store a Base64 encoded file on an EC2 instance.
Just keep in mind Base64 encoding has nothing to do with encryption!

String Manipulation

AWS CloudFormation offers multiple built-in functions to manipulate strings which are very powerful, especially when you combine them.

String Interpolation

One of the simplest string manipulations in AWS CloudFormation is interpolating a string. This can be achieved with the substitutes function Fn::Sub as in this example.

Fn::Sub: "This is my ${AWS::StackName} using Fn::Sub"

The generated string will look like similar to: This is my Stack-Name using Fn::Sub.

Fn::Sub can also be used with a mapping of variables.
Using Sub with mapping is useful for larger strings with multiple variables, as it supports the readability of the String modification and allows easier changes.

In this example the function resolves the string to the value This is my Stack-Name using Fn::Sub

Fn::Sub: 
  - "This is my ${Stack} using ${Function}"
  - Stack: 
      Ref: ${AWS::StackName}
    Function: 'Fn::Sub'

We can also combine other functions in the mapping method, such as

Fn::Sub: 
  - "This is my ${Stack} in ${Region} "
  - Stack: 
      Ref: ${AWS::StackName}
    Region:
      Fn::Select:
        - 3
        - Fn::Split:
          - ':'
          - 'arn:aws:cloudformation:ap-southeast-2:1234567890:stack/Stack/9876543210'

Here we are using the Select function in combination with the split function to retrieve the AWS Region from the AWS ARN and store it in the variable Region. The generated string will look similar to this This is my Stack-Name in ap-southeast-2.
Mapping subs can be used to automate the creation of an AWS CloudWatch Dashboard via AWS CloudFormation, as in this example on Github. They also have many other use cases.

String Concatenation

Sometimes you need to concatenate a set of values into a single string. Using the intrinsic function Fn::Join function you can append multiple values to a string with a defined delimiter. The delimiter can be any character, and you can either provide a list of comma separated values or refer to a list of comma separated values you would like to join.

Fn::Join:
  - ':'
  - - 'arn'
    - 'aws'
    - 'cloudformation'
    - 'ap-southeast-2'
    - '1234567890'
    - 'stack/Stack/9876543210'

In this example the generated string will be arn:aws:cloudformation:ap-southeast-2:1234567890:stack/Stack/9876543210.
Uses for Fn::Join range from generating a resource arn, a DNS name, or adding multiple values into a string and passing it as an input to a nested CloudFormation stack, a Lambda Function or Step Function.

String Splitting

The opposite of joining multiple values into a string is splitting them out of a string. Splitting a string into a set of strings is done with the intrinsic function Fn::Split .

Split allows you to split a string into a list of string values for further processing. This is helpful when you need to provide a list of string values as input or if you need to select just parts of a string for further processing.

Fn::Split:
  - ':'
  - "arn:aws:cloudformation:ap-southeast-2:1234567890:stack/Stack/9876543210"

In this example we generate the output ['arn','aws','cloudformation','ap-southeast-2','1234567890','stack/Stack/9876543210'], which we can use for further processing.

Indexing a value from a list

Should you need to access a value from a list use the function Fn::Select. The Select function allows you to select a single object from a list by its index.

Fn::Select:
  - '3'
  - Fn::Split:
      - ':'
      - "arn:aws:cloudformation:ap-southeast-2:1234567890:stack/Stack/9876543210"

Here we select the 3rd value from the index, which resolves to ap-southeast-2.

This function has many uses, but one I particularly find useful is to bypass the AWS CloudFormation Parameter limit. Should you reach the parameter limit, concatenate multiple parameters into a comma separated list, then use the select function within your template to select the property you need.

JSON Serialization

You need or want to generate a JSON string in your template. For example, defining a CloudWatch Dashboard body, or storing a JSON string in SSM Parameter store. Defining a JSON string can become difficult. You need to worry about character escaping in your JSON template, or you need to use JSON syntax in your YAML template. To address both, CloudFormation has the intrinsic function Fn::ToJsonString, which allow you to define the string with the same syntax as your template. Before you can use the function you need to enable the macro AWS::LanguageExtensions in your template.

Transform: 'AWS::LanguageExtensions'
...
Fn::ToJsonString:
  key1: "arn:aws:cloudformation:ap-southeast-2:1234567890:stack/Stack/9876543210"
  key2: "arn:aws:cloudformation:ap-southeast-2:9876543210:stack/Stack/1234567890"

The first line enables the intrinsic function Fn::ToJsonString. The output will look like this "{\"key1\":\"arn:aws:cloudformation:ap-southeast-2:1234567890:stack/Stack/9876543210\"},{\"key2\":\"arn:aws:cloudformation:ap-southeast-2:9876543210:stack/Stack/1234567890\"}".

Iteration

Very recently AWS CloudFormation was extended with a great new function Fn::ForEach which adds iteration to AWS CloudFormation. Iterations can be useful when you need to replicate resources on AWS. Similar to the JsonToString function, ForEach becomes available after enabling the macro AWS::LanguageExtensions.

The CloudFormation syntax looks like this

'Fn::ForEach::UniqueLoopName':
    - Identifier
    - - Value1 # Collection
      - Value2
    - 'OutputKey':
        OutputValue

You need to define a unique loop name, an identifier which will be replaced, the values you want to replace the identifier with, and OutputKey, which must include the identifier, and the OutputValue.

The iteration below produces three EC2 instances by replacing EC2InstanceId three times, with First, Second, and Third.

Fn::ForEach::Instances:
  - EC2InstanceId
  - [First, Second, Third]
  - ${EC2InstanceId}Instance:
      Type: AWS::EC2::Instance
      Properties:
...

Using iterations in your template helps you keep your CloudFormation template small and readable, but there are a few things to keep in mind. The iteration function can only be used in Conditions, Outputs, and Resources, and the default CloudFormation quotas apply to the generated final template. This includes the template size, which is not determined by the size of the template you are designing, but rather by the resolved template size. I’m going to explain this more in the next chapter.

A list of all available Instrinsic Functions is available in the official documentation.

Importing Libraries

To provide additional functionalities in CloudFormation, AWS allows you to “import libraries” (defining a transform) in your CloudFormation template.

Some CloudFormation intrinsic functions, like Select or Split, are available out of the box, but others, like Fn::ToJsonString or Fn::ForEach functions, are only available after defining the transform AWS:LanguageExtensions.

Transforms, also called macros, are a way to add additional intrinsic functions or functionalities which CloudFormation does not provide out of the box. AWS offers a few AWS managed macros, but it also provides you the ability to create and define your own macros to process the CloudFormation template. To import a macro you must define the optional Transform section and provide a list of macros you would like to import, as in this example

Transform: 
  - AWS::LanguageExtensions

As you see in the graphic, macros are executed before the CloudFormation stack gets created or updated as they are intended to modify the original CloudFormation template.

The CloudFormation user applies the CloudFormation template to AWS CloudFormation. Since the user has defined a macro in the Transform section of his template. The CloudFormation template is being send to the Lambda function associated to the macro and is returning the modified Template to AWS CloudFormation. AWS CloudFormation creates a change set and based on this change set, a new CloudFormation Stack is being created or an existing one is updated.

AWS CloudFormation allows you to define multiple macros in your template. They are processed from the most nested one, outwards to the most general macro. Other than that, macros are processed sequentially, meaning if you define more than one macro at the same location they are processed top to down.

Build your own Function(macro)

If none of the AWS-provided macros suit your needs, you can build your own macro in a few easy steps:

1. Create a Lambda Function to process an AWS CloudFormation template or template snippet.
2. Ensure your CloudFormation user has permissions to invoke the lambda function
3. Create a CloudFormation Macro, by mapping the Lambda function to a CloudFormation macro through creating a CloudFormation Stack with the definition AWS::CloudFormation::Macro.
4. Define the macro in your CloudFormation template and deploy

One reason for using your own macro could be to automatically tag resources in a CloudFormation template with mandatory AWS Tags. Other use cases could be security related.

Custom Resources

I won’t go into detail about using custom resources for stack operations, but if none of the CloudFormation functions nor your own macro will suit your use case, you may want to look into using custom resources. They allow you to invoke Lambda functions during a stack operation. The stack operation waits until the Lambda invocation has been completed.

One use case I’ve seen so far is the creation of a private CA certificate using a Lambda function, and storing it in AWS Secrets Manager; it’s more cost effective than running a Private CA on ACM.

Conditions

A standard across many programming languages is the definition of conditions. AWS CloudFormation let you define conditions to control the creation of resources or resource properties; they are supported in the Resources and Outputs section of a template.

In order to use conditions you need to define a condition by evaluating pseudo parameter or input parameter values and to use the condition where required. Due to the evaluation of parameters using the Ref function, conditions only support evaluation against strings.

Conditions are defined in the optional Conditions section of the template and always resolve to either true or false. You can use the well-known operators !And, !Or, and !Not, with conditions, but can only use the comparison operator equals for string comparison.

Parameters:
  EnableCloudFront:
    AllowedValues:
      - 'true'
      - 'false'
    Default: 'false'
    Type: String
...
Conditions:
  CloudFrontEnabled: 
    Fn::Equals:
      - !Ref EnableCloudFront
      - 'true'
...
Resources:
  CloudFront
    Type: 'AWS::CloudFront::Distribution'
    Condition: CloudFrontEnabled
...

In this example we define an input parameter EnableCloudFront, and evaluate the value of the parameter in the condition CloudFrontEnabled. The condition is true when the user has provided the string value true for the EnableCloudFront parameter. In our resource we define the Condition property to control the resource creation of the CloudFront Distribution, which is only created if the condition is true.

In this example below we define a second condition with a nested condition to control the Aliases property in our CloudFront distribution. It is worth mentioning that the exclamation marks denote key words in the conditions, and not negation (as they do in other languages).

...
Conditions:
...
  EnableCloudFrontAlias !And
    - !Condition CloudFrontEnabled
    - !Not
      - !Equals
        - !Ref 'ExternalFQDN'
        - ''

...
Resources:
  CloudFront:
...
    Properties:
      DistributionConfig:
        Aliases:
          - !If
            - EnableCloudFrontAlias
            - !Ref ExternalFQDN
            - !Ref 'AWS::NoValue'
...

The condition EnableCloudFrontAlias is true if the previous condition CloudFrontEnabled is true and the parameter input value for ExternalFQDN is not empty (not equal to the empty string). Inside the Resource property Aliases we evaluate the condition; if it is true we set the provided ExternalFQDN as an alias, if false this resource property will have no value.

Conclusion

AWS CloudFormation has implemented a wide range of software development practices into their product to help you create great CloudFormation templates. They offer some familiarity to software developers, but can be used without prior programming language knowledge. Examples for all demonstrated Software Development practices can be found in this GitHub repository.