12 Sep 2019 Tackling the Snake in the Corner: Google’s Deployment Manager
Congratulations!
You have freshly minted a set of workloads on Google Cloud Platform. Everything is running wonderfully and interest on bringing new workloads and building new software is rising. Excellent.
BUT… Things start falling apart. Infrastructure teams no longer have the well controlled provisioning of infrastructure, nothing is going through an architect stage and developers have started showing signs of a rare provisioning fever.
Symptoms are:
- You now have 78 “My First Projects” in the GCP Resource Manager.
- Data starts showing up on numerous, public facing Storage buckets.
- Instances, networks and other resources no longer share a coherent naming convention.
- Systems engineers start pulling their hair out in a panic and want to go back to the old ways.
This problem is quite solvable though and each Cloud Platform has tools to help you out.
Enter Google Deployment Manager
Any public cloud platform worth their salt will have tools to manage resources with infrastructure as code. AWS has CloudFormation, Azure has Resource Manager. For GCP we have Deployment Manager. It has some unique features that makes it potentially the most powerful of this group.
The Snake.
Each deployment/provisioning tool processes serialised data structures (JSON or YAML usually) that define a number of resources. While each solution provides different methods for defining the data structure templates, GCP’s Deployment Manager gives you a number of options for generating the YAML it requires to describe the resources. You can of course hand build YAML files (called config files) or use Jinja templates. But the one we are interested in the most is the option to define your templates using the full power of Python.
Now, when we say full power we are a little bit limited in what we are allowed to do in Deployment Manager’s Python execution environment. There are some stern warnings in the documentation about using system or network calls. Both will cause your code to fail. Those can be viewed here.
Make it Yours!
Now, so much configurability and power can have a cost. Usually having engineers crippled by decisions. So it’s important that engineering teams work together to plan what to do with all this power!
So, how might this look in practice? Put simply you would define common patterns for deployment and work from there. Starting from the top level you might design some patterns for projects in GCP. This may include patterns, schema and metadata your business may require to conform to your own policies.
For example, your requirements for each new project is for metadata labels to link your project to a cost centre, the default networks removed, and the project may be organised into a folder. The template will also bind a billing account and to cap it all off we will enable APIs for the project at the template consumer’s choosing.
In a python template this might look something like this.
the_project_template.py
import copy
import string
import random
def random_suffix(n):
'''
Create a random string for use of a suffix
takes int
returns string of length int
'''
rstring = ''
for _ in range(4):
l = random.choice(string.ascii_lowercase)
rstring = rstring + l
return rstring
def enable_apis(context, project_id):
"""
Here we process a list of apis that we wish to enable in the project
"""
resources = []
api_list = ['billing']
for api in context.properties['enabled_apis']:
depends_on = ['billing']
api_name = api + '-api'
api_list.append(api_name)
resources.append(
{
'name': api_name,
'type': 'deploymentmanager.v2.virtual.enableService',
'metadata': {
'dependsOn': depends_on
},
'properties': {
'consumerId': 'project:' + project_id,
'serviceName': api
}
}
)
return resources, api_list
def create_project(context, project_name, project_id):
"""
Here we create our initial project resource
"""
resources = [{
'name': project_name,
'type': 'cloudresourcemanager.v1.project',
'properties': {
'name': project_name,
'projectId': project_id,
'labels': {
'environment': context.properties['environment'],
'department': context.properties['department']
},
'parent': {
'id': context.properties['parent_folder'],
'type': 'folder'
}
}
},
{
'name': 'billing',
'type': 'deploymentmanager.v2.virtual.projectBillingInfo',
'properties': {
'name': 'projects/$(ref.' + project_name +'.projectId)',
'billingAccountName': 'billingAccounts/' + context.properties['billing_account']
}
}]
return resources
def remove_default_network(enabled_api_list, project_id):
"""
Generally it's a great idea to remove the default network. This helps prevent creating compute resources in strange locations/regions.
"""
# Before we take out the network we need to remove the default firewall rules
remove_networks = [
{
'name': 'remove-fw-allow-icmp',
'action': 'gcp-types/compute-beta:compute.firewalls.delete',
'metadata': {
'dependsOn': enabled_api_list
},
'properties': {
'firewall': 'default-allow-icmp',
'project': project_id
}
},
{
'name': 'remove-fw-allow-internal',
'action': 'gcp-types/compute-beta:compute.firewalls.delete',
'metadata': {
'dependsOn': enabled_api_list
},
'properties': {
'firewall': 'default-allow-internal',
'project': project_id
}
},
{
'name': 'remove-fw-allow-rdp',
'action': 'gcp-types/compute-beta:compute.firewalls.delete',
'metadata': {
'dependsOn': enabled_api_list
},
'properties': {
'firewall': 'default-allow-rdp',
'project': project_id
}
},
{
'name': 'remove-fw-allow-ssh',
'action': 'gcp-types/compute-beta:compute.firewalls.delete',
'metadata': {
'dependsOn': enabled_api_list
},
'properties': {
'firewall': 'default-allow-ssh',
'project': project_id
}
}
]
# This won't work unless all the firewall rules are gone before removing the default network.
remove_network_prereq = copy.copy(enabled_api_list)
remove_network_prereq.extend(['remove-fw-allow-icmp', 'remove-fw-allow-internal', 'remove-fw-allow-rdp', 'remove-fw-allow-ssh'])
remove_networks.append(
{
'name': 'remove-default-network',
'action': 'gcp-types/compute-beta:compute.networks.delete',
'metadata': {
'dependsOn': remove_network_prereq
},
'properties': {
'network': 'default',
'project': project_id
}
}
)
return remove_networks
def create_outputs(projectid):
'''
Create an output to expose the generate project ID.
'''
outputs = [
{
'name': "projecid"
'value': projectid
}
]
return outputs
def GenerateConfig(context):
prefix = context.properties['prefix']
suffix = random_suffix(4)
project_name = context.properties['project_name']
# We will add a suffix and prefix to create the project id which has to be globally unique.
project_id = prefix + '-' + project_name + '-' + suffix
resources = []
project = create_project(context, project_name, project_id)
enabled_apis, enabled_api_list = enable_apis(context, project_id)
removed_networks = remove_default_network(enabled_api_list, project_id)
resources.extend(project)
resources.extend(enabled_apis)
resources.extend(removed_networks)
outputs = create_outputs(project_id)
return {'resources': resources, 'outputs': outputs}
Accompanying your python is the schema file. This helps consumers of the template write a config file. It is also consumed by the Deployment Manager to test the config before applying the configuration.
the_project_template.py.schema
info:
title: Project
description: |
Creates a Project under a parent org or folder in GCP.
Attaches billing account.
Enables listed API's and removes the default network.
author: Shine Solutions
imports:
- path: project.py
properties:
project_name:
type: string
description: The Project name. The project ID will also be derived from this.
pattern: ^[a-z][a-z0-9-]{5,20}[a-z0-9]$
prefix:
type: string
description: The prefix to go before the project name eg: pre-project_name
pattern: ^[a-z]{2,4}$
billing_account:
type: string
description: |
The billing account to charge this project to.
For example, 98EA12-0B222A-82BAC0
environment:
type: string
description: production or nonproduction environment.
department:
type: string
description: department code.
enabled_apis:
type: array
items:
type: string
description: The list of API's to enable for this project.
parent_folder:
type: string
description: |
The parent folder for this project to reside under.
outputs:
projectid:
description: The id of the project created
type: string
Finally we’ll create a config that will consume our template. We can create many config yaml files to create many projects. And because of the template above they will all fall within the business requirements.
the_project_config.yaml
imports:
- path: ./a/path/project.py
name: project.py
resources:
- name: resource-name
type: project.py
properties:
project_name: data-project
prefix: shne
parent_folder: "2098738157382"
billing_account: 013220-02B488-A6CE18
department: analytics
environment: production
enabled_apis:
- compute.googleapis.com
- bigquery-json.googleapis.com
- bigqueryconnection.googleapis.com
- bigquerydatatransfer.googleapis.com
- bigquerystorage.googleapis.com
- storage-api.googleapis.com
- storage-component.googleapis.com
To create your project you simply run the following gcloud command:
gcloud deployment-manager deployments create data-project --config the_project_config.yamlA few quick notes about what is happening in the snippets above:
- To define a template in deployment manager, you have to have a function called GenerateConfig, it must have context as its only argument. This is the function that google runs in your code when you hand off the config and templates to GCP.
- The GenerateConfig must return a dictionary representation of valid YAML where the first key is resources, with the value of a list of dictionaries.
- “context” is the config file as an object (YAML config file) that gets referenced in the deployment manager command. In this case, “the_project_config.yaml”.
- Some resources in Deployment Manager are actions that can be called on certain objects. (Like the action to delete firewall rules). Use these to deviate from the standard setup that is provided by default in GCP.
- When running the above command everything that is associated with the config will be sent to Google’s deployment manager service.
Why use the Python option ?
You can absolutely use Jinja templates with Google Deployment Manager. However you will be missing out on:
- Most IDE’s already now how to work with python, usually out of the box.
- Linting and other formatting tools.
- You can write unit tests for your templates.
Although you do have a steeper learning curve when sticking with python, you will gain down the track in quality Infrastructure Code and new skills for infrastructure people. More problems solved here in the templating stage will make for easier and faster provisioning in the future.
No, REALLY make it yours!
The biggest bonus of having having a proper programming language to write the template is that you are free to write your own libraries that suit you. These libraries can hold values to mapped departments, enforce certain security requirements in code, define a minimum standard for provisioning compute instances, setup networking to automatically peer to a central network. The possibilities are endless.
Another really neat item in GDM is that they now give you the ability to register your own APIs for provisioning. These are called type providers. So if you have an outside provider for DNS or a CDN you can now register an integration with that provider and write templates to provision (for example) a distribution in your third party CDN. More info on that here.
Is your Infrastructure in Code yet ?
Some of this can seem quite daunting to IT leadership and your fellow engineers. It’s a big change moving to infrastructure as code fraught with anxiety, excitement, benefits and pitfalls.
There is no reason to rush in though. There is a lot to learn in the first few workloads you move to using Google Deployment Manager, or any of the other Infra as Code tools available. What the practice offers is that over time your teams will become super efficient at deploying old and new workloads to the cloud. As you refine your process and your infrastructure code you will quickly be able to push new ideas into being.
As always there is plenty of documentation to go through. Starting here is going to be your best bet for a GCP Deployment Manager journey.
Happy Hacking!
No Comments