AWS Migration of data lake enables massive scaling to improve energy retailer’s CX

The business challenge:

• An energy market shift from 30-minute settlements to 5-minute settlements meant a six-fold increase in the volume of data to be captured and stored for usage statistics for millions of customer accounts.
• The on-premise Oracle Exadata data store was already bursting at the seams, unreliable, and prohibitively costly.
• At times, customers browsing their account data online would experience page display delays up to 60 seconds.
• The existing platform constrained future possibilities and the client was seeking to leverage the automation, performance and reliability benefits of AWS Cloud.

The technical challenge:

The platform needed to be architected to be responsive for users, with 90% of queries across billions of rows to be resolved in less than 1 second.

The client needed to tap deeper Agile and DevOps skill with AWS, to leverage the benefits of Infrastructure-as-Code.

The solution:

• The client decided to migrate and rearchitect the entire data platform on AWS Cloud.
• The serverless consumption layer exposes data to consuming applications with extremely high responsiveness.
• AzureDevOps was introduced for version control of code assets including application code, deployment scripts, and infrastructure templates.  NewRelic and CloudWatch are used with all Lambdas.
• App code (Typescript) and Infrastructure code (Terraform) are stored together in Azure DevOps. Build of app code is automated via Azure DevOps build pipelines, and Terraform deployments are automated via Azure DevOps deployment pipelines

The benefits:

• Upon implementation of the solution, most queries were achieved within 0.5 seconds, improving the customer experience for online account management.
• Built reusable patterns for all subsequent data that needed to be exposed to other apps through APIs and microservices.
• Uplifted client’s DevOps practices and utilisation of automated tools.
• Use of AWS serverless technologies dramatically reduced operational overhead.
• Full deployment automation implements changes rapidly with significantly reduced deployment error rates.
• Adopted the use of automated testing frameworks and mocking services.

Implementation of the solution:

• Due to Shine’s previous experience in writing the original queries for the on-prem data platform, Shine’s architects and engineers were engaged to work with the client’s in-house specialists to design and deliver a Consumption Layer to expose and deliver the largest single dataset to the applications that required it.
• Key design principles were: modular, reusable scalable, secure.
• Two key elements of the project:
• 1. Creating APIs to query the data lake for the usage data for any customer service address.
• 2. Supporting a query to provide complete customer account usage data in a format suitable to support the “Power of Choice” regulation for customers to be able to compare the pricing offers of different energy retailers.
• Shine engineers also developed APIs to deliver useful data visualisations and deeper usage detail.
• Using a full DevOps model In six sprints over 12 weeks, Shine rearchitected the platform to be cloud-native with:
  – reusable patterns for all subsequent data that needed to be exposed to other apps through APIs and microservices
  – automated IaaS code patterns that could be reused for future projects.
  – serverless cloud-native AWS components, including Dynamo DB, Lambda, API Gateway, Athena, SQS and S3.
• Other technologies used include Graph QL for APIs; Terraform for IaC; Azure DevOps for CI/CD pipeline; and, NewRelic for monitoring.
• Fully automated testing included real-time monitoring of performance in production, with automated alerts if more than 10% of queries are taking more than 1 second. Actual performance averages 100ms.