Dataform test cannot see the dataset. But the dataset is actually exist.

[hao-wang]

To make the question simple. I create a dataset call test_dataset. And I can see it inside Bigquery after I created it.

Then I wrote the test of Dataform, like this:

config {

type: "test",

dataset: "test_dataset"

}

The error is:

Dataset test_dataset could not be found.

If I use type: "table", it can be found.

It should be straghtforward, but not. Can you help me with this? Thanks. 

[ruthseki]

Hi @hao-wang,

Welcome to Google Cloud Community!

The error "Dataset test_dataset could not be found" during a Dataform test execution indicates that Dataform is unable to locate the specified dataset when running the test. This typically happens due to a misunderstanding of how Dataform handles datasets during testing versus during the creation of tables/views.

Here are some approaches that you may try:

• Ensure that the test_dataset is in the same location as your Dataform project. BigQuery datasets are location-specific, and mismatched locations can cause errors.
• Verify that the service account or credentials used by Dataform have the necessary permissions to access the test_dataset. Specifically, it should have bigquery.datasets.get and bigquery.tables.get permissions.
• Double-check your Dataform configuration file dataform.json to ensure that the default location matches the location of your dataset. If there's a mismatch, update the defaultLocation field in the configuration.
• The "test" type in Dataform might have specific requirements or limitations. If it works with "table", it could indicate that "test" is not fully supported for your use case.
• Ensure that the dataset name is correctly referenced in your Dataform script. If the dataset is in a different project, you may need to include the project ID.

Feel free to explore the following documentations for more guidance:

• Dataform troubleshooting guide
• Best Practices in Dataform
• Configuring Data Pipeline Environments in Dataform
• Modern data pipeline building with BigQuery Dataform
• Dataform core reference
• Dataform core sample scripts

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[hao-wang]

Hi, Thanks for the reply!

1. Both the dataset and Dataform project are in same location (i.e. us-east1)

2. The service account 'dataform@ava-backend.iam.gserviceaccount.com' has role Bigquery Data Editor role. So it should able to access datasets in the GCP project.

3. The dataform.json file looks like:

{

"defaultDatabase": "ava-backend",

"defaultLocation": "us-east1",

"warehouse": "bigquery",

"defaultSchema": "dataform",

"assertionSchema": "dataform_assertions"

}

4. The dataset is in the same GCP project with the Dataform project

Is there anything that I m missing?

BR,

Hao