Slack notifications stopped working in GCP Alerting policy, email still working.

[Hepzi]

I noticed in slack that I'm facing an issue where slack notifications are no longer being sent through my GCP alerting policy, even though email notifications are still coming without any issues. I've been using this for months, slack alerts were working fine earlier, but today they stopped without any changes on my end.
I configured alerts for both slack and email separately in GCP Alerting policy.

Did anyone else face this issue? Is there any limit within gcp or slack that could be preventing from being sent?
Can anyone help me on this to fix?

[DamianS]

Hello @Hepzi  ,Welcome on Google Cloud Community.

1. Did you;ve checked wether you still have Google Cloud Monitoring App connected to your Slack workspace ?
2. To be sure, that channelID is correct, update your notification channel for Slack with proper Slack ChannelID. 
3. Create totally new integration for slack, then send dummy alert to verify connectivity. 

--
cheers,
Damian Sztankowski
LinkedIn medium.com Cloudskillsboost Sessionize Youtube

[Hepzi]

Hi Damian,
Thank you for your reply.  It’s still not working. Could there be something I’m missing or another area I should check?

1. I’ve verified that the Google Cloud Monitoring app is still connected to the slack workspace.
2.  I sent a test alert using curl and postman with webhook url to verify connectivity that worked as expected. I could see the test message.
3. I'm able to see the incidents in alerting console but still not getting those on slack.

[DamianS]

Are you sure that your alerting policy is using correct notification channel ? 

[Hepzi]

I'm sure the alerting policy is correct because it worked earlier. Does it have any quotas?

[Hepzi]

When I checked the cloud logging, I'm getting this message "An internal error occurred while constructing the request to the Slack API." Slack webhook url is working fine.

[Hepzi]

Please help if anyone has an idea about this.

[jayeshmahajan]

1. On GCP side. It has notification quotas, including a limit on the number of notifications sent to Slack within a certain timeframe.

• Check Monitoring Quotas in the Google Cloud Console.
  • Navigate to: IAM & Admin -> Quotas -> Filter for "Monitoring Notifications".
• If the quota has been exceeded, you may need to request an increase or wait for it to reset.
  
  2. On Slack side
• Rate Limits: Slack imposes rate limits on incoming messages via webhooks.
  • If the rate is exceeded, Slack may reject some requests temporarily.
• Message Formatting Errors: Ensure the payload adheres to Slack's message formatting requirements.
  
  Considering the logs says it an internal error occurred while constructing the request to the Slack API. You may want to look at what log entries you see on slack logs as incoming request from GCP to get the exact error message.