Author: stewartburnell

Here in Australia, we do a lot of work on Google Cloud Platform for one of the country’s largest ISPs, Telstra. Most of that work involves building data pipelines and running analytics off the back of them for their Media business unit. As you can well imagine, they generate a huge amount of data on a daily basis. We use tools like BigQuery, Cloud Dataflow and Data Studio to wrangle, manage, and understand that data. On one such project for Telstra, we saw an opportunity to delete three code repositories and finally rid ourselves of some of the headaches associated with maintaining those applications, all the while saving money on the operational costs. We were able to replace the system comprising these repos with two new Google Cloud Platform services: In this blog post, I’ll introduce you to those new services that Google have spun up, and how we were able to use them to replace our legacy applications. Who doesn’t like a good spring clean, huh?
23 January, 2018 /

Learn more >

[caption id="attachment_20558" align="alignright" width="377"]Taumata_Racer.jpg Waterslide analogy. One input, multiple outputs. Each slide represents a date partition in one table.[/caption] Do you have some data that needs to be fed into BigQuery but the output must be split between multiple destination tables? Using a Cloud Dataflow pipeline, you could define some side outputs for each destination table you need, but what happens when you want to write to date partitions in a table and you're not sure what partitions you need to write to in advance? It gets a little messy. That was the problem I encountered, but we have a solution.
05 December, 2017 /

Learn more >

I wasn’t sure what my first day at Shine would look like. I looked for some blog posts that resembled this one for some insights but I figured everyone’s experience is different. I hadn’t worked in this industry before, and my work experience at a laptop repair shop didn’t really count. The only relevant experience I had was the industry project I did in my final year of study and that turned out to be very valuable. I knew I would be thrown into the deep end and have to learn quickly. Since day one, I’ve been surrounded by great mentors, helping with code reviews, best practices to follow, great book suggestions and general insights into how this business works. Anyway, I think enough time has passed now to reflect on this year.
31 October, 2017 /

Learn more >