Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Apache Spark program not receiving messages from Google Cloud Pubsub

I have a simple Java program that reads messages from a Google Cloud Pubsub topic and prints them. It works correctly when I run the program as a standalone, but it fails to receive messages when run using spark-submit.
  1. Do programs launched using spark-submit have to follow a different structure in general? My program doesn't do any Spark related stuff as of now, but I'll be adding it later.
  2. Are there working examples of Spark + Cloud Pubsub integration? I came across a library called Apache Bahir, but is it a must to use a library like that?

    The code for my example can be found here: https://github.com/pramodbiligiri/pubsub-spark
0 2 220
2 REPLIES 2

I saw that you are already are receiving help on this stackoverflow post, so I would recommend to keep working the issue there.

Thanks. I resolved the issue. This is the answer I've posted over on Stackoverflow: https://stackoverflow.com/a/73286618/700045