Async Cloud Run design pattern

[jgeddes]

I'm looking to implement a system for initiating a variable duration process using Cloud Run. I have an API implemented as a Cloud Run service that takes an HTTP request and returns a job ID, with the actual work delegated to a separate process. The idea behind this is that the request can be made, and details of the job returned to the caller, while the slower backend processing runs asychronously.

Initially I'd planned to use another Cloud Run service to wrap the backend processing work, triggered by a PubSub notification sent by the API, however it appears that it would need to return an acknowledgement back to PubSub within a limited timeframe to prevent the message getting redelivered. Given the backend processing task may be of variable duration, it seems better suited to a Cloud Run job, but it does not seem possibly currently to trigger a job using PubSub.

Is there a recommended approach to this pattern of async backend processing with Cloud Run? Should the API (the Cloud Run service) initiate the job directly?

[robertcarlos]

Hi @jgeddes,

Welcome to Google Cloud Community!

I would suggest to use Cloud Run triggering from Pub/Sub push as:

This page describes using Pub/Sub to push messages to the endpoint of your Cloud Run service, where the messages are subsequently delivered to containers as HTTP requests.

---

Please be advised that Google recommends using push subscriptions to consume messages from a Pub/Sub topic on Cloud Run.  Pub/Sub pull subscriptions require you to monitor message delivery latency and manually scale the number of instances to maintain a healthy delivery latency. If you want to use pull subscriptions, use the CPU always allocated setting along with a number of minimum instances.

Hope this helps.

[knet]

Hi, you're right that Cloud Run jobs don't work well with PubSub yet. Do you foresee a problem with simply triggering the job directly? If your number of executions per minute is reasonably low/less than your quota, triggering the job directly should work fine. 

You could also explore using Workflows as an intermediary; there is a Workflows connector for Cloud Run jobs, and Workflows could help you handle any retry logic due to jobs failures. 

[jgeddes]

Thank you both for the recommendations, appreciate the input. After digging into this a bit more, it seems like the 10 minute timeout for PubSub acknowledgements on push subscriptions will be a blocker for us, and we don't want to go down the route of using a pull subscription, as that would require the Cloud Run service to be running continuously.

Unfortunately we'll also be unable to use Cloud Run jobs, as the workload will need to be parameterised based on the initial request.

This leaves the remaining option as a Cloud Run service for the variable-duration back-end processing, triggered by Cloud Tasks. The maximum 30 minute timeout in this case should be sufficient for our use cases, but it's slightly frustrating that it doesn't match the 60 minute timeout for the Cloud Run service itself (as noted here: https://issuetracker.google.com/issues/178424596 ).

[knet]

I'm glad you were able to find a workaround of using a service!

We're already working on parametrized jobs and pubsub support is on our roadmap, but using services is a good approach in the meantime. 

[jgeddes]

Thank you, good to know that's in the plans, in our case I think parameterised jobs would be a good fit.