An analysis pipeline that I work with depends on gcloud CLI in order to pipe data into a tool that expect standard input, but I'm struggling with handling what seem to be network failures between the VM in Google Cloud and Google Cloud Storage.
Specifically, I am seeing regular, consistent failures at varying points during the operation that say
"(gcloud.storage.cat) Download not completed. Target size=518107929774, downloaded data=99967842155", with varying target sizes and downloaded data numbers based on the size of the file and how far the download made it, of course.
I've read the API documentation at https://cloud.google.com/sdk/gcloud/reference/storage , but have not seen any information at all about retries. Am I missing something? Is there something else I'm missing causing failures of a VM running in Google Cloud downloading data from GCS?
LinkedIn
Twitter
