How to transfer files from GCP bucket to and shared drive (share drive is not on Goggle cloud)?

[rakku99]

I want to transfer the files from Google cloud storage (GCP bucket) to shared drive which is not on Google cloud. what are the right possible ways?

Regards,

Rakesh

[kolban]

Is this a one time transfer?  A scheduled transfer periodically?  Do you need low latency or batch?

[rakku99]

Yes Kolban, we need to run the batch process to transfer the files. Bath can be execute after like 6 hrs, 10 hrs or 12 hrs still depends.

[kolban]

Apologies, I should  have asked this in my last question.  What is the nature of the shared drive?  Is it a Google Drive, is it a Windows OS share, is it a Linux NFS share or something else?  Also, where is the shared drive?  Is it on a file server on premises, a drive shared from your laptop/workstation?

Do you have a preference for how you would like to bring the files in?  For example, imagine the source files were NOT on Google Cloud Storage but were on some other filesystem ... is there a copying style that you would normally use then?

[rakku99]

Thanks Kolban, This shared drive is on windows, Here we have to push the files  to shared drive using the scripts and these scripts are batch scripts which will execute after 6 hous, 12 hours. This will be Push instead of Pull (End User will not pull the files, bucket will Push the files to end user on Windows shared drive ) and how these scripts will run, Will these scripts require any compute engine to run?

[kolban]

Howdy Rakku99.  Interesting puzzle.  I've normally associated a windows machine executing a "pull" rather than a "push".  If your windows machine a user's workstation or is it an on-premises/cloud server machine?  My first pass at what I would do is to create a Linux Compute Engine and then "mount" the Windows shared drive on that Cloud based VM.  The VM then has the ability to write new files into the share.  Next, I'd look at how to read the new files from Google Cloud Storage (GCS).  My first pass at that would be to use Cloud Storage Fuse.  That technology allows a GCS bucket to appear as a mount point on the Linux file system.  Finally, I would look to have a script run at periodic intervals that would contain the logic to find the new files on GCS and copy them to the mount point of the Windows share.  This would indeed need compute running on Google Cloud ... however, I'm sensing the smallest compute possible may be sufficient as the consumption appears to be minimal.  If even that was too much, we could look at using Cloud Run and Cloud Scheduler.  In this case, the compute spins up dynamically when it is time to be of use and spins down after use.   For me, the tricky parts would be how to mount the Windows share.  I did find this article:

File transfer from linux to windows share folder on daily basis

Are there other solutions?  Yes ... very likely.  One would be to install/configure your Windows machine as an SFTP server and have the compute on VM perform an "scp" to the Windows hosted SFTP server.

[rakku99]

Thanks kolban for comments, it looks the pull mechanism. How can we achieve the push mechanism. Means we have to push the files to windows shared drive using the batch jobs periodically from gcp bucket.

[kolban]

I'm thinking the discussions in my last port were push based.   I'm seeing the Windows machine that owns the share as a passive recipient of new data.  I see the components involved being:

• Scheduling ... when should the data be copied
• Compute ... a Google Cloud hosted engine that runs the logic to perform the copying
• Cloud Storage ... the source of the data to be copied
• Windows share ... the passive target of the data to be copied

Re-read the last post I made with these in mind.  I'm sensing a Google Compute component that passively mounts the Windows share and then copies the data from GCS to the share.  Nothing need be run on the windows machine for this to happen.  I'm thinking that the windows machine will authorize the share to a Compute Engine in Google Cloud and then the compute engine will orchestrate the copies by mounting the share.

[rakku99]

Thanks a lot kolban. Here in this solution we have hosted windows share on Google compute component. If the window share in not hosted on cloud then how to achieve this tranfer?

[kolban · Accepted solution]

I'm guessing that you can perform a Windows share over the Internet.  A post here seems to say its possible although not recommended.   If you choose not to have the windows share accessed over the Internet but you still want Google Cloud to push rather than your windows machine pull, then there has to be some mechanism on your windows machine to receive the request to bring in the files when Google Cloud chooses to push them to you.   One notion that comes to mind is to install an SFTP server on a local machine which can write to the Windows Share.  An example of such a server seems to be described here.  You would then create some SSH keys and give the public key to the SFTP server and keep the private key on your Cloud Compute.  Your Cloud Compute could then perform an sftp client connection to your windows machine to push the data.

While thinking of this train of thought, I came across this article:

How to use GCP bucket data on windows file system

which seems to list a number of other stories for accessing GCS storage from windows.   These stories don't appear to be the same as copying files from the GCS buckets to a local share but instead provide stories for accessing GCS buckets as though they were windows accessible.  This is neither push nor pull (since we aren't copying) but rather access on demand.

[rakku99]

Thank you very much Kolban your precious efforts and valueable support. 

[rakku99]

Hi Kolban,

There is slight change in architecture. Now I do not have to get the date from GCP bucket. As per new requirement i have to get the data from DB/Kafka Topic and run the batch twice a day and then generate the Reports (Report generation microservice), and then transfer these reports ( csv files) to shared drive on windows Machine. The shared drive on windows Machine is on Prem and not on Cloud. The Kafka and Report generation microservice running on GKE  are on Google cloud. How can we achieve the file transfer?

Regards,

Rakesh