I am trying to figure out the source of latency in my application as I am seeing a number of un-acked messages per unit time on the dashboard for my GCP Pub/Sub subscription. My application simply publishes the consumed GCP Pub/Sub messages to a Kafka queue. I don't process the message in any way. I timed the publish to Kafka and it takes approximately 1-2ms to publish a message.
If I look at the Ack latency dashboard (found in "Metrics Explorer") which is the "...time between when Cloud Pub/Sub sends a message to a subscriber client and when Cloud Pub/Sub receives an Acknowledge request for that message." that graph has a steady latency between 25-45ms. This gives a throughput of ~25-40 messages/s. I have a number of subscriber instances but I fail to keep up with the messages that are being published to the GCP Pub/Sub topic.
I looked at my network latency and I see that the throughput of my AWS ECS service which runs the GCP Pub/Sub high-level client library is ~300-400Kbps. The n/w RX + TX is between 300-400 Kbps. Is the network contributing to the latency? Shouldn't this be much higher as GCP Pub/Sub promises 4 Gb/s network throughput per region as per https://cloud.google.com/pubsub/quotas?
My subscriber and subscription settings are mostly defaults and are as below:
Subscription:
- Message Retention = 10 minutes
- Expiration Policy = 1day
- EnableExactlyOnceDelivery = false
Subscriber:
My questions are:
I have a billing account that uses GCP Pub/Sub and I wanted to know if I need to do anything to increase the send rate from my subscription?
Is there any other setting I am missing that will help with latency? If I increase ClientCount will it lead to lower latency?
Is the network throughput issue that I see (300-400 Kbps) an issue that is on my end which I need to fix by increasing bandwidth?
Thanks in advance!
LinkedIn
Twitter
