Author: Shine Solutions Group

Shine's good friend Felipe Hoffa from Google was in Melbourne recently, and he took the time to catch up with our resident Google Developer Expert, Graham Polley. But, instead of just sitting down over a boring old coffee, they decided to take an iconic tram ride...

Intro

Post update: My good friend Lak over at Google has come up with a fifth option! He suggests using Cloud Dataprep to achieve the same. You can read his blog post about that over here. I had thought about using Dataprep, but because it actually spins up a Dataflow job under-the-hood, I decided to omit it from my list. That's because it will take a lot longer to run (the cluster needs to spin up and it issues export and import commands to BigQuery), rather than issuing a query job directly to the BigQuery API. Also, there are extra costs involved with this approach (the query itself, the Dataflow job, and a Dataprep surcharge - ouch!). But, as Lak pointed out, this would be a good solution if you want to transform your data, instead of issuing a pure SQL request. However, I'd argue that can be done directly in SQL too ;-) Not so long ago, I wrote a blog post about how you can use Google Apps Script to schedule BigQuery jobs. You can find that post right here. Go have a read of it now. I promise you'll enjoy it. The post got quite a bit of attention, and I was actually surprised that people actually take the time out to read my drivel. It's clear that BigQuery's popularity is growing fast. I'm seeing more content popping up in my feeds than ever before (mostly from me because that's all I really blog about). However, as awesome as BigQuery is, one glaring gap in its arsenal of weapons is the lack of a built-in job scheduler, or an easy way to do it outside of BigQuery. That said however, I'm pretty sure that the boffins over in Googley-woogley-world are currently working on remedying that - by either adding schedulers to Cloud Functions, or by baking something directly into the BigQuery API itself. Or maybe both? Who knows!
Shine’s TEL group was established in 2011 with the aim of publicising the great technical work that Shine does, and to raise the company’s profile as a technical thought-leader in the community through blogs, local meet up talks, and conference presentations. Every now and then (it started off as being monthly, but that was too much work), we curate all the noteworthy things that Shiners have been up to, and publish a newsletter. Read on for this month's edition.

No food reviews here I'm afraid

This year I was incredibly lucky to score a coveted ticket to YOW! in beautiful Melbourne. I was also asked to be a track host for a couple of sessions, so that was quite an honour too. This post is a whirlwind wrap-up of the conference, and only includes my favourite talks from the two day event. If you're hoping to hear detailed reviews on how the coffee/food/WiFi/venue was, then you'll be greatly disappointed (it was all great BTW).

The last time I was fortunate enough to attend AWS’s global conference, re:Invent, was three years ago in 2014. Then there were 14,000 delegates and the conference spanned just two Las Vegas hotels. Lambda was announced during Werner Vogels’ keynote and it seemed that the most in-demand sessions had “Docker” in the title.

In just three years the conference has tripled in size with 43,000 delegates attending this year spread across a campus of six Las Vegas hotels. Although not one of the biggest conferences held in Vegas, it’s obviously a significant logistical challenge. After some hiccups on the first day with the inter-venue shuttles and a venue running out of food, everything seemed to settle down and run smoothly from the start of the second day. Whether the improvement was due to human learnings of the hivemind or training of some Machine Learning algorithms is up for debate but almost certainly it was a combination of the two. No, actually, the transport still is not good and Uber is key to success.

Do you recoil in horror at the thought of running yet another mundane SQL script just so a table is automatically rebuilt for you each day in BigQuery? Can you barely remember your name first thing in the morning, let alone remember to click "Run Query" so that your boss gets the latest data refreshed in his fancy Data Studio charts, and then takes all the credit for your hard work? Well, fear not my fellow BigQuery'ians. There's a solution to this madness. It's simple. It's quick. Yes, it's Google Apps Script to the rescue. Disclaimer: all credit for this goes to the one and only Felipe Hoffa. He 'da man!
Shine’s TEL group was established in 2011 with the aim of publicising the great technical work that Shine does, and to raise the company’s profile as a technical thought-leader through blogs, local meet up talks, and conference presentations. Each month, the TEL group gather up all the awesome things that Shine folk have been getting up to in and around the community. Here’s the latest roundup from what’s been happening.
Shine’s TEL group was established in 2011 with the aim of publicising the great technical work that Shine does, and to raise the company’s profile as a technical thought-leader through blogs, local meet up talks, and conference presentations. Each month, the TEL group gather up all the awesome things that Shine folk have been getting up to in and around the community. Here’s the latest roundup from what’s been happening.