Unable to connect Cloud Data Fusion to Kafka on GCE

[mkhudia-csod]

Hello Everyone,

I am reaching out here for guidance as am unable to create a successful connection between the Cloud Data Fusion instance and the self-managed Kafka cluster running on the GCE instance privately. I would appreciate it if I can get any sort of help here. Thanks 

[josegutierrez]

This can be happening due to a Kafka limitation, where it only accepts requests to the Kafka brokers' registered listener endpoints (in this case the default hostname URLs and not the actual IPs).

To resolve this you need to re-configure your Kafka settings advertised.listeners property to the internal IPs. See :

• https://stackoverflow.com/questions/60505483/how-to-connect-to-an-internal-kafka-cluster/60521628#60...
• https://stackoverflow.com/questions/65160810/force-kafka-to-connect-brokers-through-ips-not-via-host...