Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Dataform Incremental Table Merge to also DELETE

Hello Google Community, hoping for some help here!

I've got a Dataform JavaScript file that needs to merge a source to a target using a specific condition -

If unique key is matched and field1 = value1: DELETE
If unique key is matched and field1 != value1: UPDATE
If unique key is not matched : INSERT

Currently I'm publishing the source as an incremental table with the following config -

 publish(table_name,
{
type: 'incremental',
uniqueKey: [`${tabledata['key']}`],
bigquery: {
partitionBy: "DATE(modified_pt)",
stamp_sub(current_timestamp(), interval 24 hour)"
}
}
)

This takes care of the INSERT and UPDATE operations, but I'm having to do the DELETE as a postOps. I'm not too pleased with this as the source (that populates the table publish) is generated through a complex set of CTEs and I'm having to execute the exact same CTEs again for the postOps DELETE. The only difference is the field1:value1 condition that decides whether I need to DELETE v/s UPDATE, so I'm wondering if there's a way to do all define the conditions for 3 operations just once?

Looking for something similar to the below example that I found here - https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax#merge_statement

MERGE dataset.NewArrivals T
USING
(SELECT * FROM dataset.NewArrivals WHERE warehouse <> 'warehouse #2') S
ON T
.product = S.product
WHEN MATCHED AND T
.warehouse = 'warehouse #1' THEN
  UPDATE SET quantity
= T.quantity + 20
WHEN MATCHED THEN
  DELETE

 

1 1 2,511
1 REPLY 1