Publish rate is not equal to Pull rate

amashruwala

I have 75 instances of streaming pull subscriber with the following settings:

Subscription:

- Message retention duration = 10minutes

- Expiration Policy = 1 day

- Enable Exactly Once Delivery = false

Subscriber:

- FlowControlSettings - Max Outstanding Element Count = 80, Max Byte Count = 100000000

- ClientCount = 80

I am seeing a lot of unacked messages and a lot of latency as a result and I was wondering:

1) Is my subscriber too slow?

2) Is the pull rate far too less?

For 1) I am seeing in the dashboards that the ack rate (subscription/ack_message_count) per unit time is equal to the pull rate (subscription/sent_message_count) per unit time. I have ruled out 1) as the cause of my latency

For 2) can you suggest if I need to increase the Pull rate to match the publish rate? How do I increase the pull rate to match the publish rate? Thank you!

MarvinLlamas

Hi @amashruwala,

Welcome to the Google Cloud Community!

You're right! Although the ack rate matching the pull rate indicates that your subscriber can manage the current pull rate, it doesn’t necessarily mean that the pull rate is sufficient to keep up with the publish rate. The accumulation of unacked messages and increased latency clearly point to the pull rate being insufficient.

Is my subscriber too slow?

You made the right call by examining the subscription/ack_message_count and subscription/sent_message_count. If these metrics remain roughly equal over time, it indicates that individual subscribers are not the main issue. Instead, the root cause lies in the overall pull capacity of your subscriber fleet.

2. Is the pull rate far too less?

Yes, The pull rate doesn't need to be an exact, real-time mirror of the publish rate. There will always be small changes and temporary ups and downs. The key consideration is whether the average pull rate, over a significant time period, is sufficient to prevent the accumulation and sustained growth of unacknowledged messages.

To address this you may try the following:

Increase pull rate: Increase the number of subscriber instances or adjust the flow control settings to enhance the overall pull capacity.
Optimize subscriber performance: Minimizing the time taken to process each message can help alleviate the backlog without requiring a substantial increase in the pull rate.
Monitor metrics: Leverage Google Cloud's Pub/Sub dashboards to monitor key metrics, such as unacked_message_count and oldest_unacked_message_age, to evaluate and track the performance of your system effectively.

You can refer to the following documentation, which provides information on Google Cloud's Pub/Sub. These resources offer essential guidance for monitoring and troubleshooting Pub/Sub performance, including tools and techniques to analyze subscription metrics, address pull rate issues, and enhance system efficiency through Cloud Monitoring dashboards:

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.