This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
I'm trying using the dataform-scd plugin to track a table, and at first it was working just fine. I'm not sure what changed but now I'm getting this error "Dataset projectname:source_data was not found in location EU" -- which is true you won't find the data there, my entire project is in `europe-west1` and dataform.json sets the default location `"defaultLocation": "europe-west1"`, but apparently this one javascript file is ignoring this and running on the `EU` zone, which doesn't work.
How can I find out why my script is apparently running in `EU` and why it's not respecting the defaultLocation, and is there a way to `export location = 'europe-west1'` in the js file?
Here is one of the error message outputs:
> dataform run
Compiling...
Compiled successfully.
Running...
Dataset creation failed: dataform.core_user_updates [incremental]
> create or replace table `data-project-name.dataform.core_user_updates` partition by DATE_TRUNC(updated_at, MONTH) as
>
> select * from `data-project-name.production_replica.core_user`
bigquery error: Not found: Dataset data-project-name:dataform was not found in location EU
Skipping dataset creation: ...
So you can see it's not even finding the schema/dataset! Let alone the table. And I am looking at the dataset, it's in `europe-west1` like everything else in this project. Please help; how do I get it to run dataform in a single region? (Isn't this mandatory anyway? Doesn't dataform only work with single-region datasets? I did all this data migrating to comply with this strange and annoying condition and now the script wants to run in a multi-region zone? why)
It seems like there might be an issue with the configuration or the way Dataform is executing your script.
Ensure that your Dataform configuration (dataform.json) has the correct default location set to "europe-west1". Double-check the JSON file to confirm that there are no typos or syntax errors.
Make sure that your Dataform environment is set to the correct location. You can do this by running the following command in your terminal:
If the environment is not set correctly, you might need to update it using: dataform env set <your-environment-name> --location=europe-west1
Ensure that your Dataform project configuration (dataform.json) also specifies the correct location. Check for any overrides or conflicting configurations in the project settings.
Examine the JavaScript file where the dataset is defined. Ensure that there are no hard-coded references to the wrong location. If you are explicitly setting the location in your SQL query, remove it and rely on the default location specified in the Dataform configuration.
If necessary, you can try exporting the location in your JavaScript file. At the beginning of the file, add the following line:
Hi, thanks for the reply. I have double checked all those things. dataform.json says `europe-west1` and my project is in europe-west1, and my datasets.
I tried running `dataform env` but I get this error:
$ dataform env
Dataform encountered an error: Did you mean help?
Dataform encountered an error: Unknown argument: env
and I tried adding `export const location = 'europe-west1'` but it gives this error:
I just upgraded to the most recent version of the dataform lib (2.9.0) and am using the most recent version of dataform-scd (0.5.0) so I'm not sure what to make of these different expectations in how the thing should work. I'm still not sure why my script should run in EU when every possible level of the thing is set up for europe-west1, but if this thing about exporting the location isn't working then... it sounds possible that this error is the source of the original problem too. How can I not export a variable? Am I not running javascript correctly? (I am running many many other JS-heavy apps on the same machine, and I just ran `nvm use 18` to make sure I'm running all this on a recent version.) Is the dataform CLI not running javascript correctly? Why would `export` be an unexpected token? (It's line 1 of the file.)
Update, I tried `dataform run --default-location=europe-west1` and get the same error (dataset not found in location EU). Oddly, the script runs okay when I trigger the execution from the web interface, so I guess the dataform CLI is just broken for me.
I had the identical issues and errors as @msnook including run --default-location=US not being honored The solution for me was update location in .df-credentials.json
LinkedIn
Twitter
