I have a cloud run endpoint that gets triggered by a push type pubsub subscription and it recives the user specified message thorugh publsiher,I am trying to get the message acknowledged by the acknowldgement time but the cloud run endpoint is getting triggered multiple times ,not sure if my message is getting acknowldged or not.I want to check what is the process to make sure the message gets acknolwedged with status 200ok response to pubsub in python so that there will not be any retires after one successful attempt.
Solved! Go to Solution.
Here is how to acknowledge a message in a Cloud Run endpoint triggered by a push-type Pub/Sub subscription and prevent retries:
Acknowledgment in Push Subscriptions:
For push subscriptions, where Pub/Sub sends messages to your Cloud Run service endpoint, acknowledgment occurs implicitly through an HTTP response. There's no need for the .ack()
or .nack()
methods used in pull subscriptions.
Acknowledgement with 200 OK:
To acknowledge a message successfully and prevent retries, your Cloud Run service needs to respond with an HTTP 200 OK status code. This tells Pub/Sub that the message was processed successfully and shouldn't be redelivered.
Example Code:
The following Python code demonstrates how to handle message processing and acknowledgment in a Cloud Run service:
from flask import Flask, request
app = Flask(__name__)
@app.route('/', methods=['POST'])
def index():
# Extract Pub/Sub message from request
envelope = request.get_json()
message = envelope['message']
try:
# Process message
# ...
# Acknowledge message with 200 OK
return '', 200
except Exception as e:
# Log exception
# ...
# Message not acknowledged, will be retried
return '', 500
if __name__ == '__main__':
app.run(port=8080, debug=True)
Additional Considerations:
Here is how to acknowledge a message in a Cloud Run endpoint triggered by a push-type Pub/Sub subscription and prevent retries:
Acknowledgment in Push Subscriptions:
For push subscriptions, where Pub/Sub sends messages to your Cloud Run service endpoint, acknowledgment occurs implicitly through an HTTP response. There's no need for the .ack()
or .nack()
methods used in pull subscriptions.
Acknowledgement with 200 OK:
To acknowledge a message successfully and prevent retries, your Cloud Run service needs to respond with an HTTP 200 OK status code. This tells Pub/Sub that the message was processed successfully and shouldn't be redelivered.
Example Code:
The following Python code demonstrates how to handle message processing and acknowledgment in a Cloud Run service:
from flask import Flask, request
app = Flask(__name__)
@app.route('/', methods=['POST'])
def index():
# Extract Pub/Sub message from request
envelope = request.get_json()
message = envelope['message']
try:
# Process message
# ...
# Acknowledge message with 200 OK
return '', 200
except Exception as e:
# Log exception
# ...
# Message not acknowledged, will be retried
return '', 500
if __name__ == '__main__':
app.run(port=8080, debug=True)
Additional Considerations:
I use this code but my messages are still not ACKed.
I suspect that there's a time limit for the operation done by the server for the session validity. I updated my pubsub ACK deadline and it is still NOT WORKING.
If your Google Cloud Pub/Sub messages aren't being acknowledged even though your Cloud Run endpoint returns a 200 OK status, there could be a few reasons:
Troubleshooting
Thank you for your quick response.
My situation:
I am using EVENTARC (if this changes anything).I updated my ACK deadline to 600s and my processing takes about 1min30s. I tested with an immediate 200Ok, it works. When trying in real env (with my 1min30 processing or a time.sleep), the behavior I get is that messages are received almost every ten seconds until the first ACK return. Then it stops, all the others messages are still processed (they are duplicates). It behaves exactly like the default BUT I configured the ACK deadline to be 600s, it looks like it does not even consider it.
I hope that you have a better point of view of my problem, can please give me hints on that, what other parameters need to be changed ?
Some people talk about the min retry timeout but it does not seems right : https://stackoverflow.com/a/77332635/11406744
Also, PubSub is said to be "at least once delivery", should we consider that it often happens or that IT JUST HAPPENS. I don't know if my problem is just that PubSub is doing its "at least once" on each message.
PS:
That's a problem found over and over on the web, you should PLEASE add a thorough documentation on this behavior (examples and responses cases in different scenarios and how the different parameters come into play). It is very frustrating to look for answers on stackoverlow and forums whilst the documentation could have explained it without doubt.
https://stackoverflow.com/questions/72074775/why-does-gcp-pub-sub-publish-a-message-twice
Hello I have a similar problem,
I have a Google Cloud setup where a Cloud Function is triggered whenever a new object is created in a Cloud Storage bucket. The function does some processing, and when it detects a specific file (e.g., export_complete.txt), it publishes a message (with the bucket name and directory path) to a Pub/Sub topic.
This Pub/Sub topic then triggers a Cloud Run service, which downloads and zips all files in that directory. Pubsub subscription is pull type
I have also tried an alternative approach where:
The Cloud Run service is receiving multiple POST requests for the same event, leading to duplicate processing. Some things I have tried to mitigate this:
Would really appreciate any insights or better practices to prevent multiple triggers and ensure efficient execution or an alternative approach
PS:- The only solution that has worked so far (sort of) is using a mutex lock so once a process occupies it no other request are entertained
@ms4446 I have a similar scenario where an eventarc is triggering my cloud run(push subscription). My code does some data processing tasks and it takes over 3 minutes for the process to complete before it sends the status code as 2xx. So unless my logic is executed, I cannot return the status code; hence, my cloud run gets triggered multiple times. I would appreciate a solution for this
Hi @ambeshsingh,
In your scenario where a Cloud Run service triggered by Eventarc takes more than 3 minutes to process a task, leading to multiple triggers due to the service not sending a 2xx status code in time, you can consider the following solutions:
Increase Cloud Run Timeout: If your processing time is less than 60 minutes, the first step is to increase the timeout setting of your Cloud Run service. By default, Cloud Run has a 5-minute timeout, which can be extended up to 60 minutes. This might be sufficient for your needs.
Asynchronous Processing with Pub/Sub:
Use Cloud Tasks for Task Queuing:
Implement State Management:
Optimize Processing Logic:
Cloud Run Jobs for Batch Processing:
Each of these solutions has its own trade-offs and complexities. The choice depends on your specific requirements, such as the nature of the data processing tasks, the acceptable response time, and the architectural changes you are willing to implement.
hi, @ms4446 Thank you for your response.Appreciate it. Here are some queries based on your suggestion:
1)Increase Cloud Run Timeout: The issue is not of Cloud Run being timed out but the pub/sub acknowledgment that the cloud run has to send back. Since pub/sub will wait for a max of 600 sec to receive the ack, it will retrigger the Cloud Run if it does not receive 2xx status code in the given span of 600 sec. If I modify my Cloud Run service to quickly acknowledge the Eventarc trigger by returning a 2xx status code and then publish the task details to a separate Pub/Sub topic, the issue still remains the same as the new Pub/Sub will expect the ack from the Cloud run service within 10 mins. It seems like the 10-minute timeout for PubSub acknowledgments on push subscriptions is a big blocker for us, and also we canot switch to using a pull subscription, as that would require the Cloud Run service to be running continuously.
Given your constraints and the challenges with the 10-minute acknowledgment timeout for Pub/Sub in a push subscription model, here are some revised strategies:
Immediate Acknowledgment with Asynchronous Processing:
State Management with Database or Cache:
Cloud Tasks for Deferred Processing:
Optimize Processing Time:
Hybrid Approach with Event-Driven and Scheduled Jobs:
The choice of strategy will depend on the specifics of your workload and the architectural changes you're willing to make.
@ms4446, can you help to confirm that the "Acknowledgement deadline" set on the Pubsub push subscription is actually honoured by the call? Extending the deadline doesn't seem to take any effect.
Just want to share my finding. When I created an eventarc trigger with a pubsub destination, the acknowledgement deadline of the push subscription, once created, it can't be adjusted, even though I tried modifying the push subscription directly. It seems the eventarc hardcode the ack deadline somehow, i.e. 10 secs.
When I changed to manually creating the push subscription myself (without eventarc), the acknowledgement deadline can actually be extended to a longer duration.