Fun with Serializable Functions and Dynamic Destinations in Cloud Dataflow
[caption id="attachment_20558" align="alignright" width="377"]
Waterslide analogy. One input, multiple outputs. Each slide represents a date partition in one table.[/caption]
Do you have some data that needs to be fed into BigQuery but the output must be split between multiple destination tables? Using a Cloud Dataflow pipeline, you could define some side outputs for each destination table you need, but what happens when you want to write to date partitions in a table and you're not sure what partitions you need to write to in advance? It gets a little messy. That was the problem I encountered, but we have a solution.
05 December, 2017
/ 4 Comments