Continue on Error with New-GcsObject

[craig111]

I'm trying to upload a list of folder from a file share to a Google Storage bucket. The folder contain many subfolders and over 1M items, with long file paths and some special characters thrown into the mix. After looking at calling Gsutil from within a Powershell window, I tried the cmdlet New-GCsObject to copy each folder in a for each loop. The problem with this, is that if it hits an access denied it stops uploading files. Is there a way to get round this and just log the error?

[Marramirez]

Greetings @craig111,

Welcome to the Google Cloud Community!

Take a look at: Resumable Uploads.

This allows you to resume data transfers/uploads after a communication failure has interrupted the flow of data. The gcloud CLI tool resumes upload in the gcloud storage cp and gcloud storage rsync commands when you upload data. If your upload fails/interrupted you can resume it by running the same command you used when you started the upload. You can also put the --no-clobber flag to prevent re-uploads of files that are already completed.

You can also take a look at this blog in understanding resumable uploads.

Let me know if it helped, thanks!

[craig111]

Hi Marramirez, thanks for the reply but network communication is not the issue. The problem is how the powershell cmdlet deals with errors. Firstly, it seems to include hidden items by default, where as a comparable command such as Get-ChildItem, doesn't. If it then errors out on one of these items, it then stops copying the rest of the folder, even if continue on error is enabled. Bizarrely, there's no problem copying the same data with gsutil. Although using gsutil with the powershell ISE is problematic, so it's best to a run the script via the PS command window.