Solved: Dataform Location Error - Page 2

francisatoyebi · 10-18-2023 03:30 AM

Hi all,

I have the configuration below in my dataform.json file

{

"defaultSchema": "datalake_analytics",

"assertionSchema": "dataform_assertions",

"warehouse": "bigquery",

"defaultDatabase": "cfc-2017-site-login",

"defaultLocation": "EU"

}

the "defaultLocation" is set to EU because most of our data is stored in the EU but now, I need to work on a table that is stored in "europe-west2" and this causes an error this error "Not found: Dataset domain:table_name was not found in location EU at [4:19]."
How can I specify a location for specific sqlx files to be run to avoid this error.

Thanks guys!

@ms4446 Please help and thanks for always!

ms4446

Handling datasets stored in multiple locations in GCP within Dataform can be challenging due to the current limitations of the platform. However, here are some potential workarounds and strategies you can consider:

Use the defaultLocation property: Set the defaultLocation property in the dataform.json file to the location where the majority of your datasets reside. This sets the default location for all datasets in the project. However, this approach can be limiting if you need to work with datasets in multiple locations.
Separate Projects for Each Location: Consider creating separate Dataform projects for each GCP location. This way, you can set the defaultLocation specific to each project, ensuring that each project is tailored to a specific location.
Data Transfer: If feasible, consider transferring datasets from one location to another within GCP, so they are all in the same location. This can be done using tools like gsutil or BigQuery's transfer service.

View solution in original post