Forecasting weather with BigQuery ML

Weather forecast is a complicated process. If you live in an area with lots of oscillation in weather like us in Melbourne, you should always give some chance for the weather to be different from what you see on websites.

The weather is typically forecasted by first gathering a lot of information about the atmosphere, humidity, wind, etc. and then relying on our atmospheric knowledge and a physical model to articulate changes in the near future. But due to our limited understanding of the physical model and the chaotic nature of the atmosphere, it might be unreliable.

Instead of the common approach for this, here we try to scrutinise the idea of entrusting a machine learning model for this purpose. We expect the model to look at the historical data and get a feeling of how the temperature will change in near future, let’s say tomorrow.

Getting Started with Azure Kubernetes Service

We were recently tasked with delivering a proof of concept for a large retailer to help them easily scale their Virtual Machines (VMs) and Docker containers in the Azure cloud. This meant we had to familiarise ourselves with Azure’s Kubernetes Service and we thought it would be a good opportunity to share our findings.

TEL Newsletter – October 2018

Shine’s TEL group was established in 2011, initially as a three-piece old-school-bebop Jazz combo but expanded to include a horn section during our worldwide tour of Iceland. We publicise the great technical work that Shine does, and raise the company’s profile as a technical thought-leader in the community through blogs, local meetup talks, conference presentations, and driving around shouting out of car windows. We curate all the noteworthy things that Shiners have been up to and publish a newsletter that nobody reads.

Migrating Blobstore between Projects

What is Blobstore? What is a Blob?

Like horse-drawn carriages, video rental stores, and scurvy, Blobstore is a leftover from an earlier time. It is a storage option on Google Cloud Platform (GCP) that stores objects called blobs and associates each blob with a key. It is used with Google App Engine services and allows applications to serve or get files based on an HTTP connection.

Blobstore is now superseded by Google Cloud Storage (GCS) but its usage is still possible with the actual storage in GCS, the same upload behaviour and minimal changes to the app.

In contrast to other modules in GCP, migration of Blobstore from one project to another is not straightforward. In this blog, we will investigate this migration.

Managing User Permissions in AEM

Adobe Experience Manager is designed to cater for content authoring of multiple sites by multiple content authors. Naturally, this process needs to be governed by strict Access Control Lists (ACLs) to manage who is allowed to do what at any given time. In this post, I’ll cover various approaches that can be used to manage authorizables and ACLs in AEM that should help you make a more informed decision when picking a permissions management strategy for your next project.

Running a Web Crawler in a Docker Container

Introduction

A website may have hundreds, thousands, or even millions, of public facing pages. When we are responsible for maintaining such a website, it’s impractical to traverse it manually looking for broken links. We need an automated testing tool: one which can scan the whole website and log any broken links, so we can get them fixed sooner rather than later.

In this blog, I am going to describe a web crawler project which can easily and efficiently achieve the goal. The primary technologies used in this project are Scrapy and Docker.

Using gcloud Formats and Projections in the Google Cloud Platform

Recently, I was hunting around the internet, looking for an easy way to extract an attribute of GCP resource to cross-reference while creating another resource in gcloud. I had reserved a static IP address and I wanted to use it’s IP address as the external address of a VM instance. I learnt that such a simple operation was indeed tricky, at least up until some time ago. Here’s my journey and welcome aboard!

Wrangling CloudFormation with Sceptre

Anyone who has delved into CloudFormation knows its power for describing and managing your cloud infrastructure within AWS. Likewise, if you’ve spent any time writing CloudFormation templates of any significance you’ll know that you’ll spend a lot of time duplicating sections of templates.

We always aim to reduce repetition in code so this can be a bit grating.

In this post, I hope to explore a few technologies that can help with this, primarily a tool called Sceptre from Cloudreach.

TEL Newsletter – July 2018

Shine’s TEL group was established in 2011, initially to share jam-making recipes. We publicise the great technical work that Shine does, and raise the company’s profile as a technical thought-leader in the community through blogs, local meetup talks, and conference presentations. We curate all the noteworthy things that Shiners have been up to and publish a newsletter, this very one that you’re reading right now. Hey, when you read this, whose voice do you hear in your head? Is it mine? Or yours? Everything I read is in Frank Walker from National Tiles’ voice, please help me. Read on for this edition.