It’s a simple question, often asked by project managers, data scientists, and quality engineers on every data engineering project when that first data source is ingested. How do we know the data that has been ingested into a data lake is accurate and error-free?
Updating your application to migrate from Legacy to Standard SQL isn't a straightforward process of replacing colons with dots and brackets with quotes.
Google Analytics 360 offers a way to be able to track website traffic and, by using its BigQuery integration, store the detail measures that may be useful from an analytical point of view.
Building regulatory reporting in the cloud offers many benefits, some of which are unparalleled speed, massive automatic scalability, tight security, very low initial setup cost, flexible operational costs, minimal human intervention, massive drive towards automation, and much more....
_YYYYMMDD suffix pattern just like everyone else. This works fine, but it's quite cumbersome, has some hard limits, and your SQL can quickly becomes unruly.
Then about a year ago, the BigQuery team released ingestion time partitioning. This allowed users to partition tables based on the load/arrival time of the data, or by explicitly stating the partition to load the data into (using the $ syntax). By using the _PARTITIONTIME pseudo-column, users were more easily able to craft their SQL, and save costs by only addressing the necessary partition(s). It was a major milestone for the BigQuery engineering team, and we were quick to adopt it into our data pipelines. We rejoiced and gave each other a lot of high-fives.
Shine's good friend Felipe Hoffa from Google was in Melbourne recently, and he took the time to catch up with our resident Google Developer Expert, Graham Polley. But, instead of just sitting down over a boring old coffee, they decided to take an iconic tram ride...
