Hello Google Community, hoping for some help here!
I've got a Dataform JavaScript file that needs to merge a source to a target using a specific condition -
If unique key is matched and field1 = value1: DELETE
If unique key is matched and field1 != value1: UPDATE
If unique key is not matched : INSERT
Currently I'm publishing the source as an incremental table with the following config -
publish(table_name,
{
type: 'incremental',
uniqueKey: [`${tabledata['key']}`],
bigquery: {
partitionBy: "DATE(modified_pt)",
stamp_sub(current_timestamp(), interval 24 hour)"
}
}
)
This takes care of the INSERT and UPDATE operations, but I'm having to do the DELETE as a postOps. I'm not too pleased with this as the source (that populates the table publish) is generated through a complex set of CTEs and I'm having to execute the exact same CTEs again for the postOps DELETE. The only difference is the field1:value1 condition that decides whether I need to DELETE v/s UPDATE, so I'm wondering if there's a way to do all define the conditions for 3 operations just once?
Looking for something similar to the below example that I found here - https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax#merge_statement
MERGE dataset.NewArrivals T
USING (SELECT * FROM dataset.NewArrivals WHERE warehouse <> 'warehouse #2') S
ON T.product = S.product
WHEN MATCHED AND T.warehouse = 'warehouse #1' THEN
UPDATE SET quantity = T.quantity + 20
WHEN MATCHED THEN
DELETE