07 Sep 2018 Using gcloud Formats and Projections in the Google Cloud Platform
Posted
Recently, I was hunting around the internet, looking for an easy way to extract an attribute of GCP resource to cross-reference while creating another resource in gcloud. I had reserved a static IP address and I wanted to use it’s IP address as the external address of a VM instance. I learnt that such a simple operation was indeed tricky, at least up until some time ago. Here’s my journey and welcome aboard!
Cross referencing GCP resources a very common requirement and for a production grade setup, one would leverage Deployment Manager scripts using YAML or Python. However, I was merely experimenting with some setup and was using gcloud CLI to quickly try out some steps that I had found on the web. I follow a 3-step approach – first manually set something up, then make it repeatable and configurable, and finally automate it. Writing a deployment manager template upfront would definitely be an overkill, what I was trying was a proof of concept with no promise of becoming something deployable. So automating it would outrightly derail me from trying out the configuration steps first.
“I needed a better way to introspect GCP resources within gcloud CLI itself, without depending upon any other libraries or tools…”
Seriously, Google?
Here is what I did:
- Created a VM instance in GCE and connected via SSH
- Executed some setup/configuration commands
- Perfected the commands till they worked and recorded the successful commands in a journal
- Every once in a while, I would terminate the VM and recreate it with the commands from my journal as a startup script to verify that all the commands up to that point worked
This way I was building a startup script out of tried and tested commands. If I were to made a mistake, I could quickly come up to speed through the tried-and-tested startup script. Just like we resume from where we get shot in the video game!
By moving individual commands to a startup script, I also rule-out any human error or issues with rollback. Every time I spin-up the VM instance, it is identical regardless of the machine type, region, etc. and the only way to verify the startup script is to terminate the VM instance and recreate it from time to time.
“Every time I recreated the VM instance, I hardcoded the external IP address.“
Now this may not be the most compelling of reasons, ideally we would like to have several such values parameterised. But that’s not the point here. We are far away from that stage yet. In fact, we don’t even know all the values that would make valuable parameter candidates.
I wanted to create prerequisite resources and record their attributes as variables, and simply plug the variables in subsequent gcloud statements for other resources so that the values get dynamically substituted – without ever seeing or hardcoding the actual values. So I began exploring the “gcloud describe” APIs and noticed that they return all the attributes and corresponding values of a GCP resource in JSON format. Internet made me realise that developers used a mixture of different tools like awk, grep, json formatters, etc. to extract desired attribute values. Probably GCP didn’t have solid support back in the day.
“I needed a way to make my gcloud commands repeatable & predictable…”
But not anymore, and not many developers have explored this yet. Now GCP has an elaborate suite of formats & projections to achieve this quite easily, in fact with a single gcloud command. Here I will show how I used gcloud formats and projections to help me improve my GCP infrastructure setup to be more reliable, repeatable and programmatic through gcloud CLI.
Okay, Coder… Show Me The Code!
First, let’s reserve an external static IP address:
“Yesssss! The only reason this screenshot exists, is to not let you copy-paste code!
Type the command, code-ninja. It’s all part of the plan…”
Next, let’s create a VM instance with it’s external IP as the static external address we just created. Notice that gcloud doesn’t allow us to embed a static-ip resource directly within the gcloud command for VM instance creation. So we have to hardcode the actual IP address (eek, gulp!) like this:
Aw, Snap!
We immediately notice that this is not elegant, predictable and repeatable. Not only do we have to hardcode the IP address, but we also have to expose the IP address in plain sight – yikes!
Deployment manager lets us create references, namespaces and aliases within the script, but we have not gone that far yet. Even at a basic CLI level, we would like to be able to extract the attributes of my-external-address to be programmatically used in subsequent gcloud commands and easily script-up infrastructure setup as a shell-script.
Let’s inspect my-external-address resource and explore what we get:
This is what we want to extract programmatically – without exposing the IP address.
Using gcloud Formats & Projections
Let’s format the same response into something a bit more workable. Let’s flatten the response into a map using a gcloud format:
That seems better. All the attributes are now keys and their corresponding values are, well, values! But this is still the entire config-map while we are only interested in the IP address.
Let’s use a gcloud projection to introspect the entry with key “address” which has our desired IP address as value. A projection performs exactly that – it queries the underlying flat-map to project and return the value corresponding to a key:
Bingo! That’s exactly what we’re after. However, we would like to use it without seeing it.
So let’s export it into a variable so that we may use it’s value programmatically:
Stringing together a Solution
All that looks promising so far, let’s now perform the 3 steps in tandem for a final showdown:
- Reserve a static external address
- Use gcloud format & projection to export it’s IP address into a variable
- Use the value of variable as external IP address of a VM instance
Sweet!
And we did not use awk or grep or 3rd party JSON formatting libraries…
Mission Accomplished, Top Coder!
- Our VM instance is using the static external IP that we reserved
- The external address shows that it is being used by our VM instance
- We did not expose the actual IP address anywhere in our gcloud commands
- and we successfully eliminated hardcoding for good
Our infrastructure setup is now completely detached from the actual IP address itself. If the IP address were to change, for instance, while bringing this stack up in a different region, our resources would still get connected the same way.
Knowing how to extract an attribute from GCP resource, like the IP address attribute from the static address is a handy skill. We don’t have to use Deployment Manager all the time, we can develop fairly complex infrastructure setup with humble shell scripts that are easy to learn and maintain.
Cleanup/Teardown
I always write a cleanup script to go with the setup script. It’s easy to forget intermediate resources created along the way. The best way to make the whole setup repeatable is to complement the setup automation with an accompanying teardown automation:
“The setup script is only as predictable, as repeatable the teardown script is!”
Bonus
As the project grows, so does usually the number of GCP resources too. A Managed Instance Group can spin up several VM instances and those could further create more dependencies, like GCS buckets, firewall rules, etc. It is pretty easy to get lost with too many GCP resources and keeping track of them can get out of control. Quickly being able to inspect a particular resource is a crucial ability and not being able to do that can be quite frustrating. For those of you who share a strong affinity for the CLI might get delighted that gcloud now has a powerful tool to filter GCP resources – Filters.
Filters help us narrow the results down returned by gcloud commands. We can use filters in gcloud just like we use where-clause in a SQL query. In fact they are so easy to use, that just going through the official documentation is enough. Using the gcloud filters is a breeze:
The Road Ahead
I have merely touched upon the concept of formats and projections and how to use them to introspect GCP resources in gcloud. Obviously this is barely scratching the surface and the complete suite of formats and projections is very rich with several features.
“Running three gcloud commands does not make us Gods of the CLI!“
That’s why I’ve used screenshots instead of commands in plain text that can be copied!
Formats support formatting into box, table, JSON, or YAML; use of custom delimiters and terminators, padding and flattening of the response (we used map-flattening), specification of separate formatting for various sub-sections of a response, and much more.
Projections support many types of filtering and conversions on resource entries. Conversions between table, map, and array are supported. Projections can decode and transform calendar, date, time, date-time, parts of date, work with enums, etc. Projections also categorise, group, split, organise and navigate keys/values of the properties of a resource. We can use projections to map/split/join/sort strings, arrays, etc.
The trio of Filters, Formats and Projections empowers us with the ability to slice-and-dice complex resource hierarchies, involving nested and repeating resources within resources. Clearly, much of what more can be done with these awesome tools is left to our imagination. Hope you find these tips useful to improve your productivity.
</Happy Learning!>
No Comments