Storage Transfer Service Network Routing Between Two Organizations

[Nikp]

When using Storage Transfer Service (STS) to transfer data between two separate organizations, both of which are on Google Cloud, I would like to clarify the following:

1. Does STS utilize the public internet or private network routes by default for such transfers?
2. If it uses the public internet by default, are there any options or configurations available to enable private network transfer between these organizations while using STS?

I appreciate your insights and guidance on this.

Thank you!

[ChieKo · Accepted solution]

Hi @Nikp,

By default, Storage Transfer Service (STS) uses the public internet for data transfers. However, you can configure STS to leverage private network routes for transfers between organizations within Google Cloud. This is particularly beneficial for improving security and performance.

Here are the recommendations that could help to achieve this:

• Establish VPC Peering:
• Create a VPC peering connection between the Virtual Private Clouds (VPCs) of the two organizations. VPC Peering establishes a secure, private connection between the two VPC networks.

• Configure STS Transfer Job:
• When configuring your STS transfer job, specify the source and destination buckets within the peered VPC networks.

As you work on it, it’s best to consider the following:

• Firewall Rules: Ensure firewall rules are in place within both VPCs to allow communication between the peered networks.
• Route Tables: Configure route tables within each VPC to direct traffic between the networks through the peering connection.

By following these steps, you can leverage the benefits of private network transfers for your STS operations..

I hope the above information is helpful.

[Nikp]

Thank you for the detailed information! I appreciate the clarification and recommendations.

Just to confirm, even for data transfers within Google Cloud (from one organization to another), does Storage Transfer Service (STS) still use the public internet by default unless explicitly configured with private network routes, as described?

Your insights are incredibly helpful as we work on setting this up.

[Krishna070]

Yes, by default, Storage Transfer Service (STS) uses the public internet for data transfers between different organizations in Google Cloud. To ensure the transfer does not use the public internet, you must explicitly configure private network routes by enabling Private Google Access at the subnet level for the regions involved in the transfer. This ensures the data transfer uses Google's internal network instead of the public internet.

[dheerajpanyam]

Does the same private connectivity  (obviously not with VPC peering by a hybrid solution like Cloud VPN / Interconnect) apply for transfers from onpremises to GCP for faster transfers? I didn't find any documentation on data transfer setup for onpremises to GCP.

[Krishna070]

Yes, when using Partner Interconnect on GCP, you can achieve high-performance data transfers from on-premises to Google Cloud Storage (GCS) through two primary methods:

1. POSIX Filesystem (via Agents)
2. HDFS (via Agents)

In this setup, the agents are deployed on Google Cloud VMs to facilitate seamless data transfers. This approach ensures both security and efficiency for transferring large volumes of data. Additionally, STS (Storage Transfer Service) now supports Kerberos Authentication for accessing Cloudera HDFS, further enhancing security when interacting with Hadoop environments in GCP.

For a hybrid solution like Cloud VPN or Interconnect, while VPC peering might not be the choice for direct connectivity, Partner Interconnect provides a dedicated and high-throughput path, ensuring fast, private, and reliable data transfer between on-premises systems and Google Cloud.

I hope this clarifies the setup! Let me know if you have any further questions.