google cloud monitoring with pubsub channel problem (not 100% of messages received)

[meca]

Hi,

We have set up some google cloud monitoring alert policies, and to test them put two channel :

• email alerts
• pubsub topic publishing

But the problem is that currently the pubsub publish is not reliable, indeed it receive about half of the messages, the other ones are refused by the pubsub topic with failed_precondition reason. No extra infos neither on the logs of the "stackdriver notification channel".

In the oposite, with email alerts 100% of the messages are received.

So maybe at the moment we are going to create a composer airflow dag in place of pubsub to deal with the alerts events (to write them on bigquery).

But would be interesting to understand why not all messages are published on pubsub, only a part of them.

Best regards,

[meca]

Extra infos for our problem

• below the topic metrics for cloud monitoring incindents,
  in blue the success published message, in green the refused ones with failed_precondition problem

• below the logging details of the errors from stackdriver notification channel

PS : not sure I can retrieve incidents with composer airflow, it seems incindents are not able to be retrieved by api / rest calls, so a bit stuck at the moment