Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Process sqlx file when condition is met

Hello, maybe a silly question, but I'm trying to edit current sqlx file, which is already in the pipeline and it loads data from source table, makes some transformation and save it into the destination table.

And I would like to edit the logic, to first check if data from specific date is already in the destination table if not, than continue the process, if yes, then finish the workflow without any data processing

I tried approach to declare a dependencies with custom SQL operations, but wasn't able to make it working
https://cloud.google.com/dataform/docs/dependencies

I would be really grateful for any advice to solve this issue

This is the sqlx file

 

config {
    type: "incremental",
    schema: "reporting",
    tags: ["daily"],
    bigquery: {
        partitionBy: "date"
    },
}

WITH
source AS (  
    SELECT * FROM 
      (SELECT
        SUM(total) AS sumTotal,
          `type`,
          podcastId,
          podcastTitle,
          episodeId,
          episodeTitle,
          DATE(start) as date,
        FROM ${ref(`sourcetable`)}
        WHERE ${when(incremental(), `DATE(start) = DATE_SUB('${constants.PROCESSING_CURRENT_DATE}', INTERVAL ${constants.PROCESSING_DAYS_OFFSET} DAY) AND`)}
         country IS NULL
        GROUP BY ALL
      )
)
SELECT * FROM source 

 

 Regards,
M

2 1 1,034
1 REPLY 1