How to handle assertion failure in Dataform

[tanu3012]

My purpose of this post is , How to log assertion failure in a dataform pipeline for reporting or analysis purpose. Any location where pipeline logs are stored which could be used for analysis purpose.

[kvandres · Accepted solution]

Good day @tanu3012 ,

Welcome to Google Cloud Community!

You will be able to see these failures in the Logs explorer, and you can route them depending on your preferred destination using log router, for example Cloud Storage, Pub/Sub or Bigquery. You can also route the logs to third party applications or repositories if you want. You can check these documentation to learn more: https://cloud.google.com/dataform/docs/audit-logging
https://cloud.google.com/dataform/docs/cloud-logging

Additionally, You can use looker studio for creating reports and dashboards that are easy to visualize and understand, you can connect your data from Google Cloud Storage, Bigquery, and several other data sources that can be used to create your report. You can use this link to learn more: https://cloud.google.com/looker/docs/looker-core-audit-logging

Hope this will help!

[tanu3012]

Am I able to create any consolidated report using any utility / script to read these logs. Like number of records processed , number of rejects, number of records loaded in target table.