About DataEngineer

DataEngineer · 12-16-2024

Greetings!This morning I went to update a schedule for a workflow configuration and was met with an error that my cron job schedule was no longer valid. This is despite the fact it's been functioning without issue since its inception 12 months ago.Er...

DataEngineer · 10-10-2024

This is what worked in 2.8.4 config { type: "assertion", tags: ["pos"], dependencies: ["s_o_t-member_sale_outs_units_drop_log"] } I now get this error:> Action target names cannot include '.'------------In a different sqlx, this was working in 2.8.4 ...

DataEngineer · 07-08-2024

I frequently get BigQuery log entries for these types of errors during Dataform executions. Even though this error is thrown the table execution completes successfully and is not visible in the Dataform Execution Logs. These errors are thrown seeming...

DataEngineer · 01-29-2024

I have several alerting policies set up to monitor BigQuery and they send email notifications when fired.On Wednesday Jan 24th the formatting of the email notification changed and log match conditions are no longer being included in the body of the e...

DataEngineer · 01-23-2024

When Dataform was first released as General Availability, built-in assertions (uniqueKey, nonNull,...) were actually run in a workflow as dependencies to whatever table they were attached to and run before the table was built.At some point in the rec...

DataEngineer · 05-09-2025

The Workflow Execution Logs within Dataform is for monitoring and a quick overview. GCP Logging however, is infinitely customizable. Search, filter, analyze, store,... Integrate it into pub/sub, Cloud Monitoring, Alerting,... Alerting for example, I ...

DataEngineer · 05-08-2025

Good afternoon. Workflow errors are captured automatically and can be monitored within GCP Logging: Or within the Workflow Execution Logs module in the Dataform platform.

DataEngineer · 03-24-2025

I can't help but find it comical that your post simply says you agreed with me and then told the original poster to accept your post as the "Solution". I'm assuming it just a template response but I couldn't help but laugh.

DataEngineer · 03-21-2025

It has been less than a full day, and directly pinging a community leader will not expedite the process.To answer your question, yes— 10 unique users can concurrently develop and execute the same workflows. Each workspace is effectively a repo branch...

DataEngineer · 02-20-2025

This is what I use the columns{} parameter for.https://cloud.googe.com/dataform/docs/document-tables#add_column_and_record_descriptions

My Stats

DataEngineer's Bio

Badges DataEngineer Earned

Recent Activity

Cron Job Scheduler no longer accepts range values

Moving from 2.8.4 to 3.0.4 broke my dependencies.

"<table_name> did not meet condition IF_MATCH" error on random table executions.

Cloud Alerting email notifications have changed and no longer include Log Match Conditions.

Can you make built-in assertions a dependency instead of a dependent?

Re: Error Collector in DataForm

Re: Error Collector in DataForm

Re: Dataform Fundamental question

Re: Dataform Fundamental question

Re: Best option to implement a Data Dictionary in BigQuery