Notification Channels fail case

[RyoutaGotoubp]

Hellow everyone.

I'm worried about the notification channel failing to send.

What does GCP do if the notification channel fails?

If the notification destination is at fault, is there any logic to cache the notification until the fault is resolved?

[DamianS]

Hi @RyoutaGotoubp 

Which typ of notification channel you mentioning ?

cheers,
DamianS

[RyoutaGotoubp]

Hi @DamianS 

It`s Slack and PagerDuty.

Thank you replay.

RyoutaGotoubp

[DamianS]

Hello,
Okay. For PagerDuty you must connect Stackdriver with service with proper integration key. Also, in PagerDuty,  you must configure integration type "Stackdriver". You can also create TEST alert to check connectivity. 

For Slack, you must connect app called "Google CLoud Monitoring" and add this app to desired Slack channel. Then, channel ID where you've invited this app must be add as notification channel [2]. 

[1] https://cloud.google.com/monitoring/support/notification-options#pagerduty

[2]https://cloud.google.com/monitoring/support/notification-options#slack

If you did this and still you are nto receiving alerts, check once again, whether you picked up correct notification channels during creation of alerting policy. I'm using both channels, PD as main and Slack as backup and both works fine.`

cheers,
DamianS

[RyoutaGotoubp]

Thank you for your guidance!

(SaaS = Slack and PagerDuty)

I successfully tested notifications to these SaaS.

If SaaS is not performing well,I hope the GCP to cache it.

Is notification caching logic implemented in GCP?

Or, Is it normal for notifications to be lost if the notification destination is not not performing well?

Best regard.

RyoutaGotoubp

[DamianS · Accepted solution]

Hi,

Well, GCP will not catch if any of those two are working badly and yes, it's normal. PD and Slack are 3rd party tools, so GCP works as sender, PD and Slack as receivers. There is no any additional roles here. 
What I'm suggesting to do is to create RSS feed for Slack and PD healths and get notification via email or for dedicated slack channel, every time each of them are down. But like I said, Google Cloud will not notify  you (out of the box) if PD or Slack are down in case where you using those 3rd party tools as notification channel. 
cheers,
DamianS

[RyoutaGotoubp]

Your suggestions are great!

I couldn't think of it.

It solved.

Thank you very much.