Hello, maybe a silly question, but I'm trying to edit current sqlx file, which is already in the pipeline and it loads data from source table, makes some transformation and save it into the destination table.
And I would like to edit the logic, to first check if data from specific date is already in the destination table if not, than continue the process, if yes, then finish the workflow without any data processing
I tried approach to declare a dependencies with custom SQL operations, but wasn't able to make it working
https://cloud.google.com/dataform/docs/dependencies
I would be really grateful for any advice to solve this issue
This is the sqlx file
config {
type: "incremental",
schema: "reporting",
tags: ["daily"],
bigquery: {
partitionBy: "date"
},
}
WITH
source AS (
SELECT * FROM
(SELECT
SUM(total) AS sumTotal,
`type`,
podcastId,
podcastTitle,
episodeId,
episodeTitle,
DATE(start) as date,
FROM ${ref(`sourcetable`)}
WHERE ${when(incremental(), `DATE(start) = DATE_SUB('${constants.PROCESSING_CURRENT_DATE}', INTERVAL ${constants.PROCESSING_DAYS_OFFSET} DAY) AND`)}
country IS NULL
GROUP BY ALL
)
)
SELECT * FROM source
Regards,
M