How to rate-limit push requests from Google Pub/Sub to Cloud Run?

[morpheus]

We are running Google Pub/Sub and have set it up to push messages to a Cloud Run application. We have configured max-instances on Cloud Run.

Problem we are facing: Our Cloud Run application sends queries to Cloud SQL. The SQL database gets quickly overloaded with requests from Cloud Run and cannot keep up. 

We thought configuring max-instances on Cloud Run would apply the necessary back-pressure on Pub/Sub so it won't keep pushing messages until it gets back ACK responses but that does not seem to be happening.

What we need: some setting using which we can tell Pub/Sub not to push any more messages to Cloud Run if # of unacked messages > some threshold

Can someone help us please?

[kolban]

Howdy Morpheus,

A story that may aid us here is the use of Cloud Tasks.  If I read this section of the docs, it seems that we can create "tasks" to execute and govern the rate at which they are execute and the maximum number that can run concurrently.  For example, imagine that we wanted no more than 10 requests to be sent per second.  It feels like we could use DISPATCH_RATE and set it to 10.  This would mean that no more than 10 NEW requests could be submitted per second.  However, if requests take > 1 second to process (eg. 1.5 seconds) ... then at the end of the 1st second we would have 10 already running and would immediately add another 10 giving us 20 at the start .... if we wanted to further constrain that there never be more than 10 in flight at a time, we would also seem to want to say that MAX_RUNNING should also be set to 10.  Let's mull on this and see what it can do for us.

[morpheus]

Hi Kolban,

I will look into this. I don't know if folks at Google realize this or not but it is very overwhelming for customers to decide what is the right tool for a given problem. GCP has hundreds of services and its like going to a restaurant and given a menu of 100 dishes. Good luck ordering your meal! People who work at Google know the differences between these services and what to use for a given problem. But for a newcomer this is a humonguous task to pick the right service for given problem.

M.

[kolban]

Howdy Morpheus,

I don't disagree.  Staff at Google are trying hard to explain and delineate the large array of services ... this for example is a good cheat sheet.  If you are familiar with other cloud providers, here is a mapping from Google Names to/from other vendor names.   Other good places are the overview videos on the Google YouTube channel.   And this community is a good place too.   This is a great place for new arrivals to Google Cloud and seasoned pros alike to come and meet and assist each other.  Please don't hesitate to ask questions, post comments or share opinions.