Email notification configuration for cloud data fusion pipeline failure

[iamnicks96]

We are trying to configure an email notification alert when any one of our existing cloud data fusion pipelines fails. We have tried multiple configuration options but still unable to get the notification mail, our sample configuration is as below:

We are using our Gmail username and password and SMTP protocol, still no email is received in case of pipeline failure.  Does anyone aware about this issue?

[kvandres]

Good day @iamnicks96,

Welcome to Google Cloud Community!

As stated in the documentation:

For email alerts, if your pipeline doesn't have a long-running Dataproc cluster with Simple Mail Transfer Protocol (SMTP) server, enter the mail server or service's information, such as its host and port requirements.

---

There might be some problem with your configuration, here is an example on how to set up for an email notification: https://cdap.atlassian.net/wiki/spaces/DOCS/pages/688029749/Send+Email+Post-run+Action

For more information you can check this documentation: https://cloud.google.com/data-fusion/docs/how-to/create-alerts#add-alert

This thread may also be helpful to your query: https://groups.google.com/g/cdap-user/c/B_YQVd3oH3k

Alternatively, you can try creating log based alerts using the Monitoring API and choose the emails that you want to be notified (You can configure the email by clicking manage notification channels in the step 4 of the process when setting up the log based alert). Log based alerts can also be applied as a notification in your email, if a certain message in the cloud logs appears (e.g. the pipeline succeeds, failed, etc.), you will receive an email regarding its status. Here is how you can set it up: https://cloud.google.com/data-fusion/docs/how-to/create-alerts#add-log-based-alert
You can use this link for more information regarding log based alerts: 
https://cloud.google.com/logging/docs/alerting/log-based-alerts

Hope this helps!

[iamnicks96]

Hi kvandres,

Thank you so much for answering the question. I have 2 queries of which you might be having answer

1. I tried to check if my dataproc cluster is configured with SMTP mail but I didn't found that option anywhere in the customize tab:
   

   Can you please guide us where I can check for that configuration?

2. Also I setup a log based alert with below configuration:
   

   and I intentionally failed a pipeline to check the alert:

   

   But still there is no incident shown in the alert monitor tab and no mail is generated:

   

   Can you help us a bit in resolving this issue as well.
   Any help is much appreciated.