Setting to increase GCP Pub/Sub subscription send throughput

[amashruwala]

I am trying to figure out the source of latency in my application as I am seeing a number of un-acked messages per unit time on the dashboard for my GCP Pub/Sub subscription. My application simply publishes the consumed GCP Pub/Sub messages to a Kafka queue. I don't process the message in any way. I timed the publish to Kafka and it takes approximately 1-2ms to publish a message.

If I look at the Ack latency dashboard (found in "Metrics Explorer") which is the "...time between when Cloud Pub/Sub sends a message to a subscriber client and when Cloud Pub/Sub receives an Acknowledge request for that message." that graph has a steady latency between 25-45ms. This gives a throughput of ~25-40 messages/s. I have a number of subscriber instances but I fail to keep up with the messages that are being published to the GCP Pub/Sub topic.

I looked at my network latency and I see that the throughput of my AWS ECS service which runs the GCP Pub/Sub high-level client library is ~300-400Kbps. The n/w RX + TX is between 300-400 Kbps. Is the network contributing to the latency? Shouldn't this be much higher as GCP Pub/Sub promises 4 Gb/s network throughput per region as per https://cloud.google.com/pubsub/quotas?

My subscriber and subscription settings are mostly defaults and are as below:

Subscription:

• Message Retention = 10 minutes
• Expiration Policy = 1day
• EnableExactlyOnceDelivery = false

Subscriber:

• AckDeadline = 600s

My questions are:

1. I have a billing account that uses GCP Pub/Sub and I wanted to know if I need to do anything to increase the send rate from my subscription?

2. Is there any other setting I am missing that will help with latency? If I increase ClientCount will it lead to lower latency?

3. Is the network throughput issue that I see (300-400 Kbps) an issue that is on my end which I need to fix by increasing bandwidth?

Thanks in advance!

[dawnberdan]

Hi @amashruwala,

Welcome to Google Cloud Community!

To address your questions, Here are some things to check and try in order to resolve the issue:

1. Increasing Send Rate: You may need to adjust your subscription settings to optimize throughput. Some users have reported that tweaking flow control settings, such as Max Outstanding Element Count helped reduce latency. You might also want to check if your subscription is hitting any Quota limits.
2. Reducing Latency: Increasing ClientCount could help, but it depends on how your subscribers are handling messages. If your subscribers are not keeping up with the message volume, adding more instances could improve performance. Also, adjusting AckDeadline to a lower value (instead of 600s) might help prevent unnecessary delays in message acknowledgment.
3. Network Throughput Issue: The throughput of 300-400 Kbps you’re seeing is much lower than the expected 4 Gb/s per region, which may indicate a network bottleneck within your AWS ECS service. It’s a good idea to verify if your ECS instance is allocated enough bandwidth or if there are restrictions on outbound traffic. Additionally, monitoring the subscription/delivery_latency_health_score metric could help identify potential network-related delays. You can refer to this documentation for more details. 

Here are some helpful resources to guide you in optimizing your GCP Pub/Sub subscription and reducing latency:

• Choose a subscription type
• Best practices for subscribing to a Pub/Sub topic
• Google Pub/Sub performance under low load

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.