Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Setting to increase GCP Pub/Sub subscription send throughput

I am trying to figure out the source of latency in my application as I am seeing a number of un-acked messages per unit time on the dashboard for my GCP Pub/Sub subscription. My application simply publishes the consumed GCP Pub/Sub messages to a Kafka queue. I don't process the message in any way. I timed the publish to Kafka and it takes approximately 1-2ms to publish a message.

If I look at the Ack latency dashboard (found in "Metrics Explorer") which is the "...time between when Cloud Pub/Sub sends a message to a subscriber client and when Cloud Pub/Sub receives an Acknowledge request for that message." that graph has a steady latency between 25-45ms. This gives a throughput of ~25-40 messages/s. I have a number of subscriber instances but I fail to keep up with the messages that are being published to the GCP Pub/Sub topic.

I looked at my network latency and I see that the throughput of my AWS ECS service which runs the GCP Pub/Sub high-level client library is ~300-400Kbps. The n/w RX + TX is between 300-400 Kbps. Is the network contributing to the latency? Shouldn't this be much higher as GCP Pub/Sub promises 4 Gb/s network throughput per region as per https://cloud.google.com/pubsub/quotas?

My subscriber and subscription settings are mostly defaults and are as below:

Subscription:

  • Message Retention = 10 minutes
  • Expiration Policy = 1day
  • EnableExactlyOnceDelivery = false

Subscriber:

  • AckDeadline = 600s

My questions are:

  1. I have a billing account that uses GCP Pub/Sub and I wanted to know if I need to do anything to increase the send rate from my subscription?

  2. Is there any other setting I am missing that will help with latency? If I increase ClientCount will it lead to lower latency?

  3. Is the network throughput issue that I see (300-400 Kbps) an issue that is on my end which I need to fix by increasing bandwidth?

Thanks in advance!

 
0 1 828
1 REPLY 1