AWS Firewall Manager Operational Insights

23 Oct 2024 AWS Firewall Manager Operational Insights

In a recent project, the team wanted to improve the deployment and management of Route53 Resolver DNS Firewalls across the AWS Organisation. We ultimately utilised AWS Firewall Manager and gathered lots of useful operational insights. This blog post shares those insights to assist others planning to use AWS Firewall Manager.

To follow along, a basic understanding of the following is assumed:

• AWS Organizations
• AWS CloudFormation 
• AWS Firewall Manager
• Route 53 Resolver DNS Firewall (I’ll refer to these as ‘DNS Firewalls’ for the rest of the blog)
• AWS CLI
• CI/CD pipelines

Prior to utilising AWS Firewall Manager, DNS Firewalls were provisioned using a CloudFormation template and deployed to individual accounts via a CI/CD pipeline. While this approach worked, it had its drawbacks:

• Time-consuming deployments.
• Accounts were easily missed and left with outdated configuration.
• The DNS Firewall was only applied to the default VPC in the account, meaning if a project team created an additional VPC it would not receive a DNS Firewall.
• Specific product teams required additional DNS Firewall Rules applied to their account. This was achieved via a custom bash script that we wanted to eliminate.

What did we want to achieve with AWS Firewall Manager?

• Less deployment overhead. Deploying a single CloudFormation template against 40+ accounts sucks.
• Ensure all VPCs in all accounts are automatically assigned a default DNS Firewall that the cloud engineering team manages.
• Allow project teams to edit the DNS Firewall in their dev account to allow for a degree of self-service testing and troubleshooting.
• Incremental rollout and updates of DNS Firewalls across environments.
• A fast and simple roll back procedure.
• The ability to give individual accounts additional DNS Firewall rules on top of the inherited default DNS Firewall.

Configuring AWS Firewall Manager

In this section, I’ll walk through the some of the questions we considered during the planning and design of our AWS Firewall Manager implementation. I’ll also discuss the pros and cons of the configuration decisions we made along the way.

If you haven’t already, it’s worth taking a moment and reading the AWS Firewall Manager prerequisites documentation as it will provide some helpful context.

I have created a simplified AWS Organization that I will reference and update throughout the blog.

AWS Firewall Manager Administrator Accounts

The first question I had was – should I use the default admin account for everything or use multiple delegated admin accounts? There are benefits to both approaches which I’ll outline below.

Using the Default Admin Account

In this scenario I’ve configured a single account as the default admin account, which gives it administrative scope for the entire AWS Organization. AWS recommends this account is typically a security account such as the security tooling account. To visualise this I have colour coded (in red) my example organization to reflect the security tooling account as the default admin account and its administrative scope.

Benefits:

• All of my policies are contained and managed within a single account.
• Because the default admin has full administrative scope for the entire AWS Organization, it means that you don’t need to log into the AWS Organization management account to update its administrative scope when new OU or accounts are created.
• It can simplify the configuration of your CI/CD pipelines because your code only gets deployed to a single account.

Drawbacks:

• The wide-ranging authority of the default admin account increases the potential blast radius of mistakes across the entire AWS Organization.
• Code reviews become the only safeguard against misconfigurations. For instance, if a development environment policy is inadvertently applied to a production OU, there are no built-in warnings or error alerts to notify you of the mistake. Having a Sandpit Organisation to fully test new policies can be helpful to minimise this risk.

Multiple Delegated Admin Accounts with Administrative Scope to Individual Environments

Using a delegated admin account per environment can also be a viable deployment approach, which is what I used when deploying AWS Firewall Manager. In this setup, the Infrastructure Development (Infra Dev), User Acceptance Testing (UAT), and Production (Prod) accounts each serve as delegated admin accounts. To illustrate this, I’ve colour-coded my example organization to indicate which accounts have become delegated admins and their respective administrative scopes.

Benefits

Reducing the Potential Blast Radius 

By limiting the administrative scope of our delegated admin accounts, we reduce the risk of accidentally provisioning a policy intended for a lower environment into the production environment. For example, if you use the default admin account, the potential impact is significant. A misconfigured policy meant for the development environment could inadvertently be applied to the production environment, leading to potential outages.

In contrast, by restricting the administrative scope of the delegated admin accounts, this risk is minimised. If an attempt is made to apply a policy outside its designated scope, you will most often receive an error, preventing the misconfiguration from affecting other environments. This approach helps ensure that each environment remains isolated and protected from unintended changes.

Works well with CD/CD Pipelines 

We used a CI/CD pipeline to deploy our code, with repository branches and pipelines already segregated by environments. This segregation aligned well with each administrative scope, ensuring that each environment’s deployments were isolated and could be managed independently.

Drawbacks

Additional management of administrative scope: when your administrative scope is separated by environment, you will need to update the scope configuration each time a new OU or account is added to the organization. The level of inconvenience this causes will depend on how frequently you provision new accounts or OUs. If new accounts or OUs are added frequently, managing these updates can become quite annoying.

Admin Account creation – ClickOps or AWS CLI?

Unfortunately, creating admin accounts via CloudFormation is not supported, leaving us with two options: ClickOps or AWS CLI. While either choice is acceptable, I slightly prefer using AWS CLI in conjunction with a CI/CD pipeline. This approach more closely resembles Infrastructure as Code (IaC) and offers better automation and repeatability. Additionally, the AWS Firewall Manager console has some quirks.

Quirks of the AWS Firewall Manager Console

While navigating AWS Firewall Manager through the AWS Console, I encountered several quirks that made the experience less intuitive than expected. In this section, I’ll share the aspects I found challenging and explain why they stood out. As AWS Firewall Manager evolves with updates, some points discussed here may become outdated.

Viewing Administrative Scope

I have created admin account with administrative scope to exclude a specific account, like in the below screenshot, and I want to edit this scope via the console.

By default, the console will show what the administrative scope would look like if I included accounts or OUs, which in my case is not an accurate reflect of my current configuration of excluding accounts. To see the correct configuration, I need to select the ‘Exclude only the specified accounts and organisational units, and include all others’ option from the dropdown box. This discrepancy can be very confusing, as it gives the impression that my current configuration is different from what it actually is until I realised what’s happening. This also makes it easier for misconfigurations to occur.

Updating Administrative Scope

When using the console to update the administrative scope, it seems that you cannot append configurations incrementally; you can only apply the configuration all at once. For example, I have an admin account with a scope configured to exclude a single account. I want to keep this configuration but also exclude a single (OU) aswell. However, when I add the OU, the previously excluded account is removed unless I manually reselect it while updating the configuration. I will demonstrate this below with screenshots.

Here is what the administrative scope looks like before I start, I’m only excluding a single account.

If I then click ‘edit’ I’m taken to the next screenshot.

I have edited the scope to exclude an OU from the administrative scope.

Here is a screenshot of my configuration before I apply.

After I have applied my config the OU has been applied but the account I previously listed has been removed.

I think these quirks create an unreasonable amount of testing and stress that isn’t required, for this reason I would try sticking to AWS Cli if possible.

How to create AWS Firewall Manager Admin Accounts via AWS CLI

I have modified this code for readability, so it may not function exactly as-is. If I wanted to set the security tooling account as the default admin account, this could be done by running the following command from the organization management account:

aws fms associate-admin-account --admin-account 111111111111

This following code snippet would make the development infrastructure account a delegated admin and grant it administrative scope to deploy DNS Firewalls to all development accounts and OUs across all regions. This again would need to be run from the organization management account:

aws fms put-admin-account --admin-account 666666666666 \
                          --admin-scope \
{
  "AccountScope": {
    "Accounts": [
      "333333333333",
      "666666666666",
      "999999999999"
    ],
    "AllAccountsEnabled": false,
    "ExcludeSpecifiedAccounts": false
  },
  "OrganizationalUnitScope": {
    "OrganizationalUnits": [
      "ou-xxxx-3333333",
      "ou-yyyy-666666",
      "ou-zzzz-9999999"
    ],
    "AllOrganizationalUnitsEnabled": false,
    "ExcludeSpecifiedOrganizationalUnits": false
  },
  "RegionScope": {
    "Regions": [
      ""
    ],
    "AllRegionsEnabled": true
  },
  "PolicyTypeScope": {
    "PolicyTypes": [
      "DNS_FIREWALL"
    ],
    "AllPolicyTypesEnabled": false
  }
}

At first glance, I found the AWS CLI options somewhat unintuitive. For example:

• Why do I need to specify both OUs and Account IDs? Doesn’t specifying an OU automatically include its accounts?
• What are interactions between the Accounts and ExcludeSpecifiedAccounts properties?

I’ll provide more details below, but I still highly recommend reading the AWS documentation before running these commands. (Duh)

Administrative Scope Quirks

While the AWS documentation provides a good conceptual overview of administrative scope, I felt that the distinction between deploying to OUs and individual accounts could have been explained in more detail. Below, I will take a snippet from the AWS documentation and explain what I mean.

From the AWS Documentation:

“If you want to apply policies only to specific accounts or accounts that are in specific AWS Organizations organizational units (OUs), choose Include only the specified accounts and organizational units, and then add the accounts and OUs that you want to include. Specifying an OU is the equivalent of specifying all accounts in the OU and in any of its child OUs, including any child OUs and accounts that are added at a later time.“

You need to specify both OUs and individual accounts separately in your administrative scope. When I first read the AWS documentation, I thought only including OUs would be sufficient. For example, I assumed that configuring my dev admin account with administrative scope to deploy to the dev OUs would also allow me to deploy policies to the individual dev accounts within those OUs.

However, I discovered this assumption was incorrect when I tried to deploy a policy to an individual dev account and received an error indicating that the account was out of scope. This was surprising because my impression of the documentation left me thinking that specifying an OU would allow me to target individual accounts within that OU.

In some cases, this distinction might not be significant. However, it was important for me because I needed to target specific accounts with additional, independent DNS Firewall rules. This meant I had to explicitly include those individual accounts in the administrative scope, in addition to the OUs.

This requirement adds overhead because more time is spent updating the administrative scope whenever a new account is created in an OU that needs additional DNS Firewall rules. In my case, targeting specific accounts was an exception and did not happen often and most accounts only required the default DNS Firewall rules that was applied to the entire OU.

To address this issue, one solution could be a pipeline script that automatically updates the administrative scope via AWS CLI whenever a new account is created. However, I would prefer to avoid relying on custom scripts like this if possible. Ideally, AWS would handle this scenario more intuitively, reducing the need for manual updates or additional automation.

AWS Firewall Manager Policy Insights

Now that we have set up admin accounts I will highlight snippets of CloudFormation Policy code that are noteworthy.

Creating a default DNS Firewall Policy targeted at OUs.

When creating a policy, you configure which Organizational Units (OUs) and/or individual accounts the policy applies to. In CloudFormation, this is achieved using the IncludeMap or ExcludeMap property. In my scenario, I’m setting up a default DNS Firewall policy for an entire environment. By including the OUs for that environment in the IncludeMap, any new accounts added to those OUs automatically fall within the policy scope and receive the default DNS firewall settings.

In my code, I’ve established a mapping (dev -> defaultOus -> ous) that contains all the OUs intended to receive the default DNS Firewall policy for that environment. The policy references this mapping value, ensuring that any changes or additions to the listed OUs are automatically reflected in the policy’s scope without requiring manual updates.

  defaultVpcDnsFirewallPolicy:
    Type: AWS::FMS::POLICY
    Properties:
      DeleteAllPolicyResources: true
      ExcludeResourceTags: false
      PolicyName: !Sub ${env}-default-vpc-dns-firewall-policy
      ResourcesCleanUp: true
      IncludeMap:
        ORGUNIT: !FindInMap [ !Ref env, defaultOus, ous ]
      RemediationEnabled: true
      ResourceType: AWS::EC2::VPC
      SecurityServicePolicyData:
        Type: DNS_FIREWALL
        ManagedServiceData: !Sub '{"type":"DNS_FIREWALL","preProcessRuleGroups":["ruleGroupId":"${defaultDnsRuleGroup}", "priority":99]}'
      Tags:
        - Key: Protect
          Value: Foundation
        - Key: StackId
          Value: !Ref AWS::StackId
        - Key: Env
          Value: !Ref env

Mappings:
  dev:
    allowedDomains:
      defaultAllowedDomains:
        - 'www.shinesolutions.com'
        - '*.shinesolutions.com'
    defaultOus:
      ous:
        - ou-xxxx-3333333 # Security Dev
        - ou-yyyy-666666 # Infra Dev 
        - ou-zzzz-9999999 # Workload Dev

It’s important to note that the administrative scope of the admin account deploying this policy takes precedence over anything listed in the IncludeMap or ExcludeMap. If the IncludeMap specifies OUs or accounts outside of the admin account scope, an error message will occur. This reinforces the importance of carefully defining and limiting the administrative scope, as discussed earlier. By restricting the administrative scope appropriately, such as only allowing deployment within specific environments or OUs, you can prevent accidental application of policies intended for one environment into another (e.g., dev policies affecting prod).

This helps reduces the potential blast radius of policy deployments.

Creating a Additional Specific DNS Firewall Policy for an Individual Account

After deploying the default DNS Firewall to all VPCs within our OUs, we can address another common scenario. Some individual accounts may require additional privileges beyond what is provided by the default DNS Firewall. In such cases, we can target these specific accounts with a policy that grants the necessary additional privileges.

For example, recently, a new ‘Project X’ account has been added to each workload OU, and they require access to ‘www.google.com‘ and all its subdomains. I’ve updated my example organization to include these Project X accounts.

I will update my mappings to include the additional allowed domains required by Project X (dev -> allowedDomains -> projectXAccountAllowedDomains) and the Account ID for Project X (dev -> accounts -> projectXAccountId).

Then I create a new policy specifically for Project X and in the IncludeMap property I reference the Project X Account ID mapping. This kind of configuration means that Project X will always receive the default DNS Firewall Rule while having the flexibility of having more rules applied to it’s account.

  projectXAccountVpcDnsFirewallPolicy:
    Type: AWS::FMS::POLICY
    Properties:
      DeleteAllPolicyResources: true
      ExcludeResourceTags: false
      PolicyName: !Sub ${env}-projectX-vpc-dns-firewall-policy
      ResourcesCleanUp: true
      IncludeMap:
        ACCOUNT: 
          - !FindInMap [ !Ref env, accounts, projectXAccountId ]
      RemediationEnabled: true
      ResourceType: AWS::EC2::VPC
      SecurityServicePolicyData:
        Type: DNS_FIREWALL
        ManagedServiceData: !Sub '{"type":"DNS_FIREWALL","preProcessRuleGroups":["ruleGroupId":"${projectXAccountDnsRuleGroup}", "priority":1]}'
      Tags:
        - Key: Protect
          Value: Foundation
        - Key: StackId
          Value: !Ref AWS::StackId
        - Key: Env
          Value: !Ref env 

Mappings:
  dev:
    allowedDomains:
      defaultAllowedDomains:
        - 'www.shinesolutions.com'
        - '*.shinesolutions.com'
      projectXAccountAllowedDomains:
        - 'www.google.com'
        - '*.google.com'
    defaultOus:
      ous:
        - ou-xxxx-3333333 # Security Dev
        - ou-yyyy-666666 # Infra Dev 
        - ou-zzzz-9999999 # Workload Dev
    accounts:
      projectXAccountId: 
        - 777777111111 # Project X Acc Dev

Allowing all subdomains does not grant access to the root domain itself. For instance, if I specify ‘*.shinesolutions.com’ in the DNS Firewall rules, it will not allow resolution for ‘shinesolutions.com’ itself. While this may be obvious to many readers, I’ve included it here for clarity, especially for first-time users encountering this concept.

Policies need be used in conjunction with other security controls

AWS Firewall Manager is effective for applying and automatically remediating protections for resources. However, it’s important to remember that they must be used with other security controls such as Service Control Policies (SCPs) to prevent project teams from disabling or bypassing these protections. For instance, I’m enforcing a default DNS Firewall across all accounts, restricting access to a list of allowed domains. Without adequate SCPs in place, there’s nothing preventing product teams from creating new a DNS Firewall rule group with a lower priority, making my DNS Firewall policy redundant.

Additionally, overly strict SCPs can inadvertently create more work. For example, the default DNS Firewall is managed by the cloud engineering team. If a product team needs to test new domains within their development account, a too-strict SCP could lead to frequent requests for the cloud engineering team to update their DNS Firewall settings. Implementing an SCP that allows product teams to add their own DNS Firewall rules specifically in their development accounts can provide them with a degree of self-service for testing and troubleshooting. However, it’s important to balance this approach with the security standards and policies of your organization.

The Policies Rule Groups can’t be viewed outside of the admin account it was created in.

Here’s an annoying scenario I encountered: after applying my DNS Firewall Policy, I logged into an account that had received the policy and found that while the policy successfully attached the rule groups to my VPCs, I couldn’t see the specific rules included in those rule groups. This creates a problem because project teams are unable to directly view the DNS Firewall rules from their accounts. This limitation hinders product teams from quickly troubleshooting errors. As a result, the cloud engineering team had to create a Confluence page that linked to the repository containing all the rule groups.

Should the ‘ResourcesCleanUp’ property be enabled or disabled?

This property determines whether protections applied to a resource from your policy will be removed if the resource becomes out of scope of the policy. Enabling or disabling this feature carries both benefits and drawbacks, which I’ll outline below.

Enabling ResourcesCleanUp:

Enabling this property can potentially reduce conflicts during the deployment of future policies. For example, if ResourcesCleanUp is disabled and I remove a policy that applied a DNS Firewall to a specific account, the DNS Firewall will remain enabled in that account even though the policy has been removed. In the future, if I attempt to apply a new DNS Firewall policy to that account, conflicts can arise if the new policy is too similar to the old one. For instance, they might share the same name or priority numbers in the rule groups. In such cases, I would have to manually log into the account and remove the old protections before I could apply my new policy.

Automatically removing old configurations also means you won’t have potentially outdated security protections scattered throughout your organization. Additionally, it’s important to note that in many cases, your product admin teams may not have the permissions to remove these protections themselves, which can result in additional work for you.

This feature also allows you to quickly and automatically roll back new policies. For example, if you accidentally apply a policy to your production account, you can simply fix the misconfiguration in your CloudFormation template and redeploy the template. If this property were disabled, you would have to manually delete resources in a production account, which can be a risky and time-consuming process.

Disabling ResourcesCleanUp:

One small benefit of disabling resourcesCleanup is that if you remove a policy from an account and forget to apply a new one, the resources in that account will still retain some protections. This can serve as a safety net, ensuring that resources are not left entirely unprotected in the interim.

Out of Compliance Resources

AWS Firewall Manager has a handy console screen that lets you check if any accounts are out of compliance from their policies. However, it’s easy to miss issues since you won’t be automatically notified if a policy fails to deploy or remediate resources. To ensure you are promptly informed, you will need to set up an SNS Topic to receive email notifications for this functionality.

AWS Firewall Manager is expensive!

At the time of writing, it costs $100 USD per policy, per region, per month to use AWS Firewall Manager. Using my example organization I’ve done some napkin maths to estimate my monthly costs. Keep in mind that this estimate covers only the AWS Firewall Manager costs; there may be additional fees for the services that the policies enable. For a detailed breakdown, refer to the AWS pricing page here.

Finally, should you use the AWS Firewall manager?

AWS Firewall Manager is a great tool to centrally manage and enforce policies within your organization and I recommend using it when cheaper alternatives aren’t available or don’t meet your requirements. In my case, slight changes in my situation might have led me to choose a different solution instead of AWS Firewall Manager. However, there is no definitive answer on whether you should use a particular service. The best approach is to gather as much information and as many criteria as possible regarding your specific circumstances and evaluate how well the service meets your requirements. I hope this blog helps to inform your decision 🙂

Please note that all opinions expressed in this blog were applicable at the time of writing and may evolve or become outdated as the AWS Firewall Manager service is updated.