What is Managed kafka in GCP under the hood?

[dheerajpanyam]

It would be good if the GCP documentation has a diagram that shows the internals of Managed Kafka in GCP  atleast at a high level. I understand there is PSC,  load balancers etc but would be nice if there is a diagram.

[diannemcm · Accepted solution]

Hi @dheerajpanyam,

Welcome to Google Cloud Community!

Here’s the Network architecture for Google Cloud Managed Service for Apache Kafka:

Flexible networking

The service makes a cluster accessible from any VPC securely. This includes access from multiple VPCs, projects, and regions.

To configure networking for a cluster, you provide the set of subnets where the cluster is accessible. The service provisions private IP addresses for the bootstrap servers and brokers in each subnet. It also sets up private Cloud DNS with URLs for each IP address. The bootstrap servers have a load balancer, so there is a single bootstrap URL per cluster. The URLs are the same across all VPCs so client configurations can be consistent across environments.

This level of flexibility is achieved thanks to Private Service Connect (PSC). Each IP address allocated for a cluster requires a PSC endpoint. The endpoints are provisioned automatically.

Note that Google Cloud Managed Service for Apache Kafka cluster in GCP is under Pre-GA.

Other relevant references for Google Cloud Managed Service for Apache Kafka can be found here:

• Overview
• Cluster management
• Pricing
• Authentication via Open-Source-Kafka-API
• Authentication via managed service

I hope this helps!

[dheerajpanyam]

Thanks so much @diannemcm  for the detailed explanation. We are planning to use this service in production based on when it would become GA. Would it be safe to assume a 3 month timeline to be GA as it happens with other GCP services?

[diannemcm]

Hi @dheerajpanyam,

Some services can take up to three months to completely launch; with Google Cloud Managed Service for Apache Kafka, there is no given timeline. What I can provide you is the release note where all updates about this product are being posted. 

I hope the above information is helpful.

[KirTitievsky]

@dheerajpanyam How did it go with this service? If there is anything we can do to help, let me know. I work on the Managed Service for Apache Kafka  team.