I was adding a new source in Datastream whose base db is in Postgres I’ve setup the pipeline where the destination is through cloud storage and then the dataflow job sends the data to BQ because we wanted to logs separately. When I’m backfilling the data/when the new updates are coming it’s flowing into the staging dataset but not the main dataset, the table is created but with 0 data, essentially the merge operation is not happening. Any idea why this could be happening?
LinkedIn
Twitter
