How can I get data from a folder in Sharepoint (XLSX, CSV files, for example) and insert it into GCP

[gustavribeiro]

I need to perform a batch process to upload this structured data (aka tables) to cloud storage (or bigquery), but I can't get access even though I'm the page owner in SharePoint. Has anyone done something similar? PS: I can't do it via Airflow either.

[yan-hic]

Was on the same boat trying to automate pulling from OneDrive in Python. Doable but quite complex.
You may want to consider pushing from within instead of pulling from external i.e. use Power Automate to  save to Google Drive, from which BigQuery can read from.

MS even made it easy and provides a template for that: SharePoint Online files to Google Drive Files | Microsoft Power Automate 

[gustavribeiro]

I can't use Google Drive too 😞 We can't have access. A few months ago I tried to do it by power automate but the connection with BigQuery was in beta version, so it doesnt work well at that time. I'll check again

[Mongine]

If you need to copy special file types (XLSX, CSV files ........ etc) automatically and directly from SharePoint to GCP,  A cloud migration tool like Gs Richcopy360 or AvePoint will do it easily and quickly,  google the 2 tools

[Kanika_Panwar]

To retrieve the files, we can either use PowerShell to access and download them or leverage the Postman REST API. By using Postman, the files can be downloaded from the folder via the API endpoint: https://contoso.sharepoint.com/sites/yoursite/_api/web/GetFolderByServerRelativeUrl('/sites/yoursite...'). Additionally, a script can be written within Postman to automate uploading these files to Google Cloud Platform (GCP), ensuring a seamless process.