I once again had the opportunity to head to Snowflake’s world tour in Sydney, and thought I would take the time to summarise some of the things I took away from the conference. If you're interested in what I thought last year, you can read...
In my previous post, I wrote about how I used AWS DMS to create an Oracle database replica in Aurora PostgreSQL. I touched on how my tests uncovered slow query performance and how using the query plan management extension helped me to achieve comparable execution...
Recently, I got involved in an cloud application uplift project. As part of the project we wanted to use an Aurora PostgreSQL database to offload all read activities from a RDS Oracle Database Standard Edition.Since this was such an exciting and educational experience, I wanted...
It’s a simple question, often asked by project managers, data scientists, and quality engineers on every data engineering project when that first data source is ingested. How do we know the data that has been ingested into a data lake is accurate and error-free?
For a long time, relational databases have been the answer to every data persistence need (whether they really suited the solution or not). But in the previous decade, there has been an explosion of different types of database engines. In this post I'm going to...
AppSync integrates seamlessly with a DynamoDB database. And as demonstrated in my previous article, AWS Amplify CLI can create the DynamoDB tables and generate the API CloudFormation infrastructure from an annotated GraphQL schema. However, using a relational data source with AppSync is more complex as...
Shine's very own Pablo Caif will be rocking the stage at the very first YOW! Data conference in Sydney. The conference will be running over two days (22-23 Sep) and is focused big data, analytics, and machine learning. Pablo will give his presentation on Google BigQuery,...
With the current move to cloud computing, the need to scale applications presents itself as a challenge for storing data. If you are using a traditional relational database you may find yourself working on a complex policy for distributing your database load across multiple database instances. This solution will often present a lot of problems and probably won’t be great at elastically scaling.As an alternative you could consider a cloud-based NoSQL database. Over the past few weeks I have been analysing a few such offerings, each of which promises to scale as your application grows, without requiring you to think about how you might distribute the data and load.