I have a requirement, where I will be getting a encrypted zip file in GCS bucket. I need to decrypt and unzip that. Inside the unzipped file, I will be getting ~40 JSON files which I need to ingest(Raw, no transformation) into BigQuery table.
The issue is, I will be getting the encrypted zip every 15 minutes. When I try to decrypt, unzip and load the json files into BQ, it is taking ~10 minutes (including resource provisioning). I dont want to take the risk of 5 minutes buffer time.
Please suggest if any workaround is there to bring down the resource provisioning time or any other way I can solve this requirement
LinkedIn
Twitter
