No native solution SFTP to GCS? For real?

[Dovile]

I am exploring options for copying/moving some files from SFTP to GCS, Google Cloud Storage.

Does GCP for real don't have an out-of-the box solution for that? Something that wouldn't require some custom code and compute engine or some siloed pub/sub operations.

I was looking at https://cloud.google.com/integration-connectors/docs/connectors/sftp/configure but is is not clear how to manage that. My use case in in analytics space and there was no plan to use appdev services. 

Thank you

[robertcarlos]

Hi @Dovile,

Welcome to Google Cloud Community!

You may want to check out this article on SFTPGo access to GCS via SFTP as this might be the one that you are looking for.

Also, here's the Github link for the documentation and steps on how to use SFTPGo.

Hope this helps.

[avindia]

I am looking for transfer files periodically from my GCP Project GCS Storage bucket to Third Party SFTP server. Please advice if there is out of the box service available from GCP . Does SFTP Connector ?

[avindia]

I received perfect solution from Staff Madhuvandhini as below 

To transfer from GCS to SFTP, below steps can be followed

1)GCS connector can be used to download file

https://cloud.google.com/integration-connectors/docs/connectors/cloudstorage/configure#dwld

2)Use SFTP connector to upload file using upload action

https://cloud.google.com/integration-connectors/docs/connectors/sftp/configure#actions-exp

ok

[tm235]

You can use this https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/transfer/sftp_to_gc... SFTP to GCS operator in cloud composer

[ugion12]

If you are comfortable with deploying code with Cloud Functions you've got the exact instructions in this Medium article someone has been helpful to publish: transfer SFTP to BigQuery via Cloud Storage (skip the part where it further transfers the file from GCS to BigQuery). Then again you can use a service provider like (URL Removed by Staff) that does it automatically for you for a monthly fixed price. Or some other third-party that charges by use.

Honestly deploying the SFTP to GCS function is a great way to get started with Google Cloud Functions if you're up for it. Getting familiar with it will open many opportunities also for the future to automate data flows that you need to.

[dchiesa1]

Another option: 

just use a Debian server, and configure it properly to use GCS as the backing store, via gcsfuse. 

https://github.com/DinoChiesa/sftp-server-in-gce

[JeffThornTech]

Hi Dovile, 
We offer Out-of-the-box solutions (no coding required) for SFTP and web/HTTP transfers to Google Cloud (or any cloud) on the Google Cloud Marketplace. https://console.cloud.google.com/marketplace/browse?hl=en&inv=1&invt=AbrOzQ&q=Thorn%20Technologies%2...

Feel free to Contact us and mention this post. We'd be happy to help you with a free trial and a private offer. https://thorntech.com/contact-us/

I hope this helps. 

Cheers,
Jeff

[dchiesa1]

Nice. Jeff , regarding

"Out-of-the-box solutions (no coding required) for SFTP and web/HTTP transfers to Google Cloud (or any cloud) "

I want to better understand the phrase "SFTP transfers to Google Cloud".   In your solution, does the SFTP endpoint reside inside the customer VPC in Google Cloud?  Or, does the data transfer happen to an SFTP endpoint that resides outside of a customer's Google Cloud VPC, and then data gets transferred again, into the Google Cloud VPC? 

do you have a diagram or further explanation? 

 thanks.

[JeffThornTech]

Hi @dchiesa1, 

We offer two products - SFTP Gateway is an SFTP service that streams data directly to cloud storage locations, like Google Cloud Storage. StorageLink is a brandable web interface to private cloud storage locations. It is like a "private dropbox" backed by your cloud storage bucket. 

To answer your question directly - the SFTP endpoint resides inside the customer VPC. Ideally, it points to a Network Load Balancer for HA. 

Both products run as Virtual Machine images inside a customer's VPC. The instances are ephemeral - data streams directly to cloud storage and is never stored on the VM instances themselves. This is a MAJOR differentiator for us. We are not a SaaS service, and we don't have access to any customer data. Because instances don't hold any state, they easily scale horizontally to support high-availability configurations for Enterprise workloads. We also have a containerized version (not available on the public marketplace just yet). 

Both applications have an easy-to-use Admin Dashboard that makes configuration and administration a breeze. They integrate with any OIDC Identity providers or LDAP directory services so you can tap into your existing user directory. 

More information about our products can be found here - https://thorntech.com/products
https://help.thorntech.com/ 

Here are our marketplace listings. We have three pricing tiers Standard (up to 5 sftp accounts), Pro (up to 100 sftp accounts), and Enterprise (1000+ sftp accounts). 
https://console.cloud.google.com/marketplace/browse?hl=en&inv=1&invt=AbrOzQ&q=Thorn%20Technologies%2...

Hope this helps! I am happy to connect if you want to chat further! 

Jeff

[mikeholden]

I'm Jeff's marketing guy. I'm jumping here with our blog post on Google Cloud SFTP, which explains more about how the SFTP Gateway works on Google Cloud. We also have a white paper, SFTP Gateway for the Enterprise, which has multiple diagrams and other details.