I'm trying using the dataform-scd plugin to track a table, and at first it was working just fine. I'm not sure what changed but now I'm getting this error "Dataset projectname:source_data was not found in location EU" -- which is true you won't find the data there, my entire project is in `europe-west1` and dataform.json sets the default location `"defaultLocation": "europe-west1"`, but apparently this one javascript file is ignoring this and running on the `EU` zone, which doesn't work.
How can I find out why my script is apparently running in `EU` and why it's not respecting the defaultLocation, and is there a way to `export location = 'europe-west1'` in the js file?
Here is one of the error message outputs:
> dataform run
Compiling...
Compiled successfully.
Running...
Dataset creation failed: dataform.core_user_updates [incremental]
> create or replace table `data-project-name.dataform.core_user_updates` partition by DATE_TRUNC(updated_at, MONTH) as
>
> select * from `data-project-name.production_replica.core_user`
bigquery error: Not found: Dataset data-project-name:dataform was not found in location EU
Skipping dataset creation: ...
So you can see it's not even finding the schema/dataset! Let alone the table. And I am looking at the dataset, it's in `europe-west1` like everything else in this project. Please help; how do I get it to run dataform in a single region? (Isn't this mandatory anyway? Doesn't dataform only work with single-region datasets? I did all this data migrating to comply with this strange and annoying condition and now the script wants to run in a multi-region zone? why)
LinkedIn
Twitter
