Canary deployments with Spinnaker and Kubernetes – Part 1

04 Oct 2019 Canary deployments with Spinnaker and Kubernetes – Part 1

“Move fast and break things” – an oft-cited Mark Zuckerberg quote – is seen as the catchcry of modern software deployment practises. To be competitive in fast-moving industries where software is central to a companies success, developers and software engineers have had to find ways to push their software faster, while at the same time de-risking the process of doing so.

Naturally many of the current best practises in this area have been spawned from and developed within the sphere of the tech giants in Silicon Valley. One of the methods that have become common amongst more advanced software shops in recent years is canary deployments.

What is a canary deployment? Simply put it’s a deployment technique that involves deploying a new version of a service and gradually shifting more traffic towards the new version of the service. During this shift of traffic, you are continually testing certain metrics against the new service and then rolling back the release if a certain threshold is not met.

While this sounds like a relatively simple concept, the nuance and complexity of implementation are far from it. Up until the last couple of years the technology that allowed reliable canary deployments was more or less out of reach for all but the biggest tech companies. The advent of Kubernetes, containers, service meshes and more advanced continuous deployment tools has allowed us mere mortals more of an opportunity to access these advanced techniques.

In this two-part post, I’m going to explore the setup of Spinnaker on AWS EKS. Spinnaker is a continuous deployment platform released by Netflix in 2015. It was spawned from another tool Netflix released in 2012 called Asgard. Spinnaker is an extremely powerful opensource platform for managing software deployments. It supports many different platforms and deployment methodologies, for the purpose of this blog we are going to run it on top of a managed Kubernetes cluster in AWS. We will use Prometheus for collecting metrics that we will base our canary deployment pipeline on and Jenkins (along with Docker Hub) for building and hosting our container.

Our end goal will be to deploy an extremely simple golang web service that will test for increased latency post-deployment and rollback the deployment if we exceed a certain threshold.

For clarification as we are dealing with relatively advanced concepts I won’t really be going into detail about what Kubernetes and containers are, there is a plethora of good information available online about these technologies. I’ll also mention here, this is a pure how-to article, with the aim of giving you a platform to test out canary deployments on Spinnaker. Spinnaker is relatively difficult to get up and running, so if I can save some other people from having to discover it’s quirks I’ll be happy.

Also a short disclaimer. The resources spun up in this tutorial are not within the free tier and will carry a cost.

Let’s get started!

Pre-requisites

1) If you don’t have a Kubernetes cluster and you want to follow through with the CloudFormation samples I’ve created, you’ll need to install Sceptre to run these samples. The samples were created with a relatively old version of Sceptre (1.4.0) – It can be installed using “pip install -Iv sceptre==1.4.0”, with instructions for operating here https://pypi.org/project/sceptre/1.4.0/

2) You will need the aws-cli along with a valid profile for your AWS account on your machine. – https://docs.aws.amazon.com/cli/latest/userguide/install-linux-al2017.html

3) Kubectl. This is the command-line tool for managing Kubernetes clusters. If you don’t have it already follow the simple instructions here for installation https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html

4) aws-iam-authenticator. Allows you to connect to Kubernetes via kubectl using AWS IAM permissions. Instructions for installing are here https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html

Creating an EKS cluster

For starters, if you don’t have an EKS cluster in your account, you can find some example templates I have created here. You will want to go ahead and make a copy of the “prod” folder and fill in the YAML templates contained within with parameters relevant to your AWS account. A couple of important points, 1) Make sure the profile in the config.yaml file matches the AWS profile you have set up for your AWS CLI. 2) You may need to change the amiId parameter to match a relevant EKS optimised in AMI in your region if you aren’t in ap-southeast-2. I think the rest of the parameters are self-explanatory but go through them to make sure you have everything up to date.

Creating the stack is simply a matter of running “sceptre launch-env <env>” from the root of the ekssample directory. In my case, it was “sceptre launch-env prod”. If everything goes okay you can head to your EKS console and you should see a cluster called “ekssample”

Just a note at this point.  Our cluster’s Kubernetes endpoint is by default publically available. In a non-demo scenario, I wouldn’t normally do this. Instead, it would be best practise to make your API private and access it through a bastion host or some similar mechanism.

Configuring our tools

First up let’s configure kubectl and test it against our new cluster. You are going to want to have your AWS profile environment variable set if you don’t already (export AWS_PROFILE=profile). Run an “aws s3 ls” from the command to make sure that’s all good. Next run

aws eks update-kubeconfig --region ap-southeast-2 --name ekssample

Change your region to whatever region you are operating in. After running this we should be able to run “kubectl get namespaces” to get a list of namespaces in your cluster.

At this point in time, if you run “kubectl get nodes” your worker nodes aren’t joined to the cluster. We have to apply a config map to our cluster that allows our worker nodes to connect. You can find an example here. In that file, edit “rolearn” to match the arn of the role attached to your worker nodes. Save the file and run

kubectl apply -f aws-auth-cm.yaml

Run “kubectl get nodes –watch” and wait for your nodes to reach ready state.

Installing Helm

We’ll be using Helm to install Jenkins and Prometheus. Helm is a tool that aids in providing consistent templates (charts) for running services on Kubernetes. Without it, our lives would be a lot more difficult!

To install Helm run

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh

chmod +x get_helm.sh

./get_helm.sh

Helm uses a service called Tiller to co-ordinate deployments inside Kubernetes. To enable this we will first need to set up a service account for it to use.

You can find an example of the template here.

Download that file and run “kubectl apply -f tillerserviceaccount.yaml“

After that is done we can run “helm init –service-account tiller” to install the Tiller service in our account.

Installing Jenkins

With Helm configured, we now simply run “helm install –name stable-jenkins stable/jenkins” to install the latest version of Jenkins into our cluster. This is the only chart we are installing that will create an external load balancer inside your AWS account.

After running this command, run “kubectl get pods –watch” and you will see when Jenkins is up and running.

Once Jenkins is up and running execute the following two commands

printf $(kubectl get secret --namespace default stable-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

and

export SERVICE_IP=$(kubectl get svc --namespace default stable-jenkins --template \ "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}") && echo http://$SERVICE_IP:8080/login

The first command will return your administrator password for Jenkins. The second will return the login URL for your Jenkins load balancer. Grab both and head to the URL and log in (username will be admin).

Configuring Jenkins

First up, let’s configure our Jenkins slave container to allow us to build a docker image.

Navigate to “Manage Jenkins” > “Configure System”

Navigate down to the “Cloud” section, under “Kubernetes” you will find a default Kubernetes pod template. Make the following configuration changes to this template.

• Name: slave
• Labels: slave
• Usage: Only build jobs with label expressions matching this node

Under the container template, change these settings

• Docker image: ninech/jnlp-slave-with-docker
• Working directory: /home/jenkins/agent

Create a host path volume with the following settings (Volumes > Host Path Volume):

• Host path: /var/run/docker.sock
• Mount path: /var/run/docker.sock

With the JVM being a bit memory heavy, I also like to go into the advanced settings for the container and bump the limit and request memory up to 1024Mi, then the limit and request CPU up to 1024m.

Save this config and head back to “Manage Jenkins” > “Configure Global Security”

Navigate down to Agents and enable TCP Agent Protocol/3. We wouldn’t normally do this in a production environment, but the container we are using, in this case, requires it.

Now is a good time to update all our plugins in Jenkins (particularly important for the Kubernetes plugin). Go to “Manage Plugins” and run all available updates, allow Jenkins to restart once this is complete.

Next up let’s install the golang plugin so we can build our project. Under “Manage Jenkins” > “Manage Plugins”. Go to the “Available” tab, search for “golang” and you’ll find the Go plugin, install this.

Once installed, go to “Manage Jenkins” > “Global Tool Configuration” and add a Go tool installation as per the screenshot below. This will allow us to download Go during our build.

As mentioned earlier, we are going to use Docker Hub to store our container. If you don’t have an account, go ahead and create one. Note down your credentials.

From the home screen of Jenkins, select “Credentials” > “System” > “Global credentials”. On the left-hand panel of the page select “Add Credentials”. Select “Username with password” for kind, add your Docker Hub username and password. For ID enter “dockerhub”. Save this.

We are now going to set up our build job. My build is based on the application here: https://github.com/densikat-shinesolutions/simple-web-server. You can clone that yourself and put it in your own Github repo. Within the Jenkinsfile for that project, make sure you adjust the docker.build and docker.image parameters to reflect your Docker Hub username and repo that you will store the container in. In my case they are “densikatshine” and “simple-web-server” respectively, “simple-web-server” is a simple public repo I have created in my Docker Hub account.

To create the build job, go to the Jenkins home screen, select “New Item”, enter “simple-web-server” for the item name and select “Multibranch Pipeline”.

Under “Branch Sources” add a git repository and enter your clone address for your project. In my case it’s https://github.com/densikat-shinesolutions/simple-web-server.git

Save this. A build on the master branch should automatically kick-off. All things going well, at the end of the build your container will build and be pushed to Docker Hub.

Phew! So far, so good for part 1 of this tutorial. Just before we finish, we will run the Prometheus Helm chart in preparation for part 2.

Installing Prometheus

Let’s install Prometheus. There is a Helm chart we will use to install this. Run the following

helm install --name stable-promethus stable/prometheus

Take note of the installation notes of how to connect to your Promethus install, we won’t need that for this tutorial, but it’s good to jump into and have a look around.

Wrapping up

So we’ve finally got to the end of part 1. If you run a “kubectl get pods” you should see something similar to the screenshot below

In part 2, I’ll be back to run through the process of installing and configuring Spinnaker and finally setting up our Canary deployment!