I once again had the opportunity to head to Snowflake’s world tour in Sydney, and thought I would take the time to summarise some of the things I took away from the conference. If you're interested in what I thought last year, you can read...
It's no secret that when it comes to building data platforms, I've spent a fair amount of time building cloud provider native solutions, both in AWS and GCP. So when I saw that Snowflake had a conference showcasing all their latest and greatest features, I...
Recently, I got involved in an cloud application uplift project. As part of the project we wanted to use an Aurora PostgreSQL database to offload all read activities from a RDS Oracle Database Standard Edition.Since this was such an exciting and educational experience, I wanted...
It’s a simple question, often asked by project managers, data scientists, and quality engineers on every data engineering project when that first data source is ingested. How do we know the data that has been ingested into a data lake is accurate and error-free?
Building regulatory reporting in the cloud offers many benefits, some of which are unparalleled speed, massive automatic scalability, tight security, very low initial setup cost, flexible operational costs, minimal human intervention, massive drive towards automation, and much more....
Shine's very own Pablo Caif will be rocking the stage at the very first YOW! Data conference in Sydney. The conference will be running over two days (22-23 Sep) and is focused big data, analytics, and machine learning. Pablo will give his presentation on Google BigQuery,...
One of the projects that I'm currently working on is developing a solution whereby millions of rows per hour are streamed real-time into Google BigQuery. This data is then available for immediate analysis by the business. The business likes this. It's an extremely interesting, yet challenging project. And we are always looking for ways of improving our streaming infrastructure.As I explained in a previous blog post, the data/rows that we stream to BigQuery are ad-impressions, which are generated by an ad-server (Google DFP). This was a great accomplishment in its own right, especially after optimising our architecture and adding Redis into the mix. Using Redis added robustness, and stability to our infrastructure. But – there is always a but – we still need to denormalise the data before analysing it.In this blog post I'll talk about how you can use Google Cloud Pub/Sub to denormalize your data in real-time before performing analysis on it.[embed]https://www.youtube.com/watch?v=3QHNsE9Y6Qs[/embed]Video of the recent 'Big Data the Cloud Way' presentation by Shine's Graham Polley and Shane Neubauer is now available.In it, Graham and Shane talk about how they used both the Google and Amazon cloud stacks at Telstra to query massive quantities of data...