Struggles with the bits

I decided to explore restructuring some of our ETL processes that feed data to BigQuery in order to have them stop saving data files locally. Currently, we pull data from multiple sources, apply some transformations to them, save local copies of the transformations as well as push the latter to BigQuery. In doing this, we essentially make it possible for our on-premise data silos to tap into those locally saved files.