Australian Census enabled by migration to the cloud

August 11, 2022

For the 2021 Census, the Australian Bureau of Statistics (ABS) made a significant advancement in its journey to move more of its services and systems into the cloud. 

The Census Digital Service leveraged AWS Cloud scalability to enable a target of 350 census form submissions per second at peak or 1.26 million responses per hour on Census day. Other pivotal cloud projects enabled the execution of a successful Census, including an Operations Insight platform developed in partnership with Shine and AWS Professional Services.

The project created the first cloud-based data lake for the ABS to store data on how the Census was progressing. This system ingested operations data from multiple data sources and produced approximately 100 reports showing the progress of Census operations and enabling the ABS to make numerous critical, real-time decisions.

The aim of the Operations Insights project was that all Census staff could monitor 2021 Census operations with near real-time data and make informed decisions quickly. Some specific aims for the Operations Insights tool were:

The tool had three processes that are typical of a data lake:

• – The ingestion of the raw operational data (most of which was not designed for reporting).
•  -Transform (or derive) the data it into a format that can be easily understood for reporting.
• – Visualise and report the data. We used the Microsoft Power BI tool for end users to view these reports themselves.

The data in most reports refreshed at least once an hour with some refreshing every five minutes. This streamlined approach was a large improvement from the spreadsheets used for monitoring in the 2016 Census, which were circulated once a day and included only a fraction of the breadth and depth of metrics in the 2021 reports.

The new Operations Insights tool for 2021 significantly improved the capability for operations staff to make decisions. It provided them with near real-time data, tailored reports and visualisations

This detailed case study explores Shine’s role in migrating the data to the cloud and creating the data lake with more than 1200 light and heavy data transformations.