How to overcome cloud deadlines/timeouts for async long tasks

[plasdeniability]

Hello, I hope you are well. As the title says, my problem is short, but it's leaving me out of ideas anyway:

I use Cloud Run and, upon a given user request, the application sends a message to a Pub/Sub topic. Linked to this topic, there is a PUSH subscription to pass the message to an application route (also on Cloud Run) to run the given long task asynchronously from the initial requester (user).

The problem is that Pub/Sub has a 10 minute deadline (600 seconds) and Cloud Run has a 60 minute timeout, but sometimes a single task can take a lot of time to run, exceeding all the limits.

In that case, what would be the ideal approach? Be it code approach or infrastructure approach.

The only requirement is that these tasks should run asynchronously from the requesting user's point of view and should be executed only once. I was thinking about having a Compute Engine MIG PULL'ing from the subscription, but the problem is that I fear that it will result in tasks that could be executed more than once in two or more of these instances.

I saw Cloud Workflows and Cloud Batch, but I don't think they fit in this case of asynchronous task initiated at user request, do they? 

[kolban]

This looks a good match for the question along with answers ... lets see if it answers your question ...

https://stackoverflow.com/questions/66157881/gcp-cloud-pub-sub-push-subscription-max-ack-deadline-no...

[Joevanie]

Try using Compute Engine with a PULL subscription from Pub/Sub and use message ordering of Pub/Sub. Message ordering ensures that messages are delivered in order and only one message per ordering key is processed at a time.