The business challenge:
- An energy market shift from 30-minute settlements to 5-minute settlements meant a six-fold increase in the volume of data to be captured and stored for usage statistics for millions of customer accounts.
- The on-premise Oracle Exadata data store was already bursting at the seams, unreliable, and prohibitively costly.
- At times, customers browsing their account data online would experience page display delays up to 60 seconds.
- The existing platform constrained future possibilities.
The technical challenge:
The platform needed to be architected to be responsive for users, with 90% of queries across billions of rows to be resolved in less than 1 second. Upon implementation of the solution, most queries were achieved within 0.5 seconds.
- The client decided to migrate and rearchitect the entire data platform on AWS Cloud.
- The serverless consumption layer exposes data to consuming applications with extremely high responsiveness.
- Improved the customer experience for online account management.
- Built reusable patterns for all subsequent data that needed to be exposed to other apps through APIs and microservices.
- Uplifted client’s DevOps practices and utilisation of automated tools.
- Use of AWS serverless technologies dramatically reduced operational overhead.
- Full deployment automation implements changes rapidly with significantly reduced deployment error rates.
- Adopted the use of automated testing frameworks and mocking services.
Implementation of the solution:
- Due to Shine’s previous experience in writing the original queries for the on-prem data platform, Shine’s architects and engineers were engaged to work with the client’s in-house specialists to design and deliver a Consumption Layer to expose and deliver the largest single dataset to the applications that required it.
- Key design principles were: modular, reusable scalable, secure.
- Two key elements of the project:
- 1. Creating APIs to query the data lake for the usage data for any customer service address.
- 2. Supporting a query to provide complete customer account usage data in a format suitable to support the “Power of Choice” regulation for customers to be able to compare the pricing offers of different energy retailers.
- Shine engineers also developed APIs to deliver useful data visualisations and deeper usage detail.
- Using a full DevOps model In six sprints over 12 weeks, Shine rearchitected the platform to be cloud-native with:
– reusable patterns for all subsequent data that needed to be exposed to other apps through APIs and microservices
– automated IaaS code patterns that could be reused for future projects.
– serverless cloud-native AWS components, including Dynamo DB, Lambda, API Gateway, Athena, SQS and S3.
- Other technologies used include Graph QL for APIs; Terraform for IaC; Azure DevOps for CI/CD pipeline; and, NewRelic for monitoring.
- Fully automated testing included real-time monitoring of performance in production, with automated alerts if more than 10% of queries are taking more than 1 second. Actual performance averages 100ms.