GCP subscribers not performing optimally

[amsh]

I'm currently managing multiple GCP subscribers connected to a single GCP subscription, but despite numerous attempts at optimization, the subscribers consistently fail to keep up with the incoming message rate. Here's an overview of my latest setup:

Infrastructure:

i) 25 AWS ECS instances (each with 1 vCPU and 1GB RAM) running C# GCP subscribers using Streaming Pull with high level client library.

ii) Approximately 100 streaming pull connections (according to GCP metrics).

GCP Subscription:

i) Message Retention: 10 minutes

ii) Expiration policy: 1 day

iii) AckDeadline: 600s

iv) EnableExactlyOnceDelivery: false

GCP Subscriber:

i) AckDeadline: 600s

ii) AckExtensionWindow: 590s

iii) FlowControl - maxMessageCount: 5000, maxOutstandingByteCount: 5000 * 700

According to AWS ECS metrics, combined network throughput across all 25 instances averages around 600–700 kbps, with occasional peaks up to 2 Mbps.

I was under the impression that the streaming pull client supports 10 Mbps per client, yet I’m seeing far lower throughput across all instances.

Scaling up to 75 instances didn’t improve throughput or reduce message backlog.

The topic's publish rate is approximately 20,000 messages per second, but even with acking without doing anything with messages, the subscribers can’t handle that rate.

GCP Pub/Sub metrics show frequent modifyAckDeadline calls, but there are no expired messages.

I've also tried using default settings with similar results. The dashboard reports an average acknowledgment latency of 25ms, and the ack rate closely matches the message send rate.

Given the current setup and performance data, how can I configure or optimize my subscribers to handle the input rate of 20k messages per second more effectively?

Any insights or suggestions would be greatly appreciated. Thanks in advance!

[caryna]

Hi @amsh,

Welcome to Google Cloud Community!

Check the list below for possible solutions to the issue:

• Examine your AWS network performance and try upgrading AWS instance type.
• Re-evaluate your flow control settings to prevent your subscriber from being overwhelmed by messages.
• Consider reducing your AckDeadline, a high value might delay the redelivery of failed messages in Pub/Sub.
• Keep monitoring Cloud Pub/Sub and AWS ECS to ensure it’s performing well and to catch any issues like messages or delays.
• You may also refer to Troubleshooting a pull subscription which provides common symptoms and potential causes. It highlights what key metrics you need to monitor, client configuration issues and also discusses “not enough subscribers” and others.

If the issue persists, please contact Google Cloud Support. When reaching out, include detailed information and relevant screenshots of the errors you’ve encountered. This will assist them in diagnosing and resolving your issue more efficiently.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.