Deadlettering is not working.

[ankit-pradhan]

Hi,

Deadlettering is not working . Followed below steps. Please tell us a solution. 

1. Created a new google service account (say : svc-pubsub@myproject.iam.gserviceaccount.com) which is not attached service account.
2. Created topics and subscriptions (my-topic , my-deadletter-topic, my-subscription , my-deadletter-subscription)
3. my-deadletter-topic is configured as deadlettering policy for my-subscription.
4. Given below roles to service account on topics (my-topic , my-deadletter-topic).
   4.1 ) "roles/pubsub.publisher",
   4.2 ) "roles/pubsub.subscriber"
   4.3 ) "roles/pubsub.viewer"
5. Given below roles to service account on subscriptions (my-subscription , my-deadletter-subscription).
   5.1) "roles/pubsub.subscriber"
   5.2) "roles/pubsub.viewer"
6. deadlettering is enabled on my-subscription with 10 maxAttempts.
7. Started publishing message on my-topic from my-publisher-app. (Using google client library)
8. Started stream pull from subscription (my-subscription) in my-subscriber-app. (Using google client library)
9. I am sending nack always from subscriber in my-subscriber-app. (Using google client library)
10. Messages are not getting written on my-deadletter-topic.

[ms4446]

From the steps you've followed, it looks like you've set up dead lettering correctly in Pub/Sub. However, there are a few things you might want to check to troubleshoot the problem:

1. Ensure proper permissions: Confirm that the service account (svc-pubsub@myproject.iam.gserviceaccount.com) you've created has the necessary permissions. The service account should have the pubsub.subscriber role for the original subscription and the pubsub.publisher role for the dead letter topic. You've already set these, but it might be worth double checking.

2. Check dead lettering settings: Ensure that dead lettering is enabled correctly for the subscription in question. You should have set a DeadLetterPolicy with the deadLetterTopic set to the name of the dead letter topic, and maxDeliveryAttempts set to the maximum number of delivery attempts. In your case, you've set maxDeliveryAttempts to 10, which should be fine.

3. Check your application logic: Ensure that your subscriber application is correctly sending a negative acknowledgement (nack) after receiving a message. This should trigger Pub/Sub to consider the message undeliverable after the specified number of delivery attempts.

4. Verify message delivery: Check if the messages are actually being published to the topic and being delivered to the subscriber. There might be a problem with the publishing or subscribing that is preventing messages from being delivered.

5. Ensure correct topic and subscription naming: Make sure you are using fully qualified topic and subscription names, in the format projects/{project-id}/topics/{topic-name} and projects/{project-id}/subscriptions/{subscription-name} respectively.

6. Check for Pub/Sub client library issues: There might be an issue with the Google Cloud Pub/Sub client library you are using. Check if there are any known issues or bugs related to dead lettering in the version you are using.

If all these steps are correctly implemented and you're still experiencing issues, it might be helpful to look at the logs to see if there are any error messages that can help pinpoint the problem.

[ankit-pradhan]

Hi @ms4446 ,

I check more on this and looks like I created user managed google service account(svc-pubsub@myproject.iam.gserviceaccount.com) with necessary roles as mentioned above.
But I don't have google managed google service account (service-{projectId}@gcp-sa-pubsub.iam.gserviceaccount.com) with role "Cloud Pub/Sub Service Agent" in my project. I think this is created by google itself for dead-lettering.

I think google Pubsub uses this service account for dead-lettering instead of "svc-pubsub@myproject.iam.gserviceaccount.com" which I created with all the roles and permission.
Can you confirm the same ?
If yes how can we configure it so that my service account as mentioned above is used for dead-lettering.
Link : https://cloud.google.com/pubsub/docs/handling-failures#grant_forwarding_permissions

[ms4446 · Accepted solution]

Yes, you're correct. Google Cloud Pub/Sub uses a service account created and managed by Google, the service-{projectId}@gcp-sa-pubsub.iam.gserviceaccount.com service account, to forward undeliverable messages from a subscription to a dead-letter topic. This is mentioned in the official Google Cloud documentation you've referred to.

The Google-managed service account is the one that needs permission to publish to the dead-letter topic. This is because the Pub/Sub service, not your application or the service account your application uses, is what actually moves the message from the subscription to the dead-letter topic.

To allow the Google-managed service account to publish to the dead-letter topic, you need to grant the roles/pubsub.publisher role to the service-{projectId}@gcp-sa-pubsub.iam.gserviceaccount.com service account on the dead-letter topic.

Unfortunately, you can't configure Pub/Sub to use your own service account for dead-lettering. The process of forwarding messages to the dead-letter topic is managed by the Pub/Sub service, not by your application, so it needs to use the service account that is associated with the Pub/Sub service.

Here's how you can grant the required permissions:

1. Go to the Google Cloud Console.
2. Go to IAM & Admin.
3. Click on IAM.
4. Click on ADD.
5. In the "New members" field, add the Google-managed service account (service-{projectId}@gcp-sa-pubsub.iam.gserviceaccount.com).
6. In the "Role" dropdown, select "Pub/Sub Publisher".
7. Click on SAVE.

After granting the necessary permissions to the Google-managed service account, the dead-lettering feature should work as expected.

[ankit-pradhan]

Thanks @ms4446