Solved: BigQuery datasets broken - Page 2

Karsten189 · 12-23-2024 05:15 AM

Hello community,

I am moving the BQ datasets in all projects across my organization from the single-region europe-west3 into the multi-region "EU". This has been working fine for all projects except one.

Usually I created a new replica per dataset, made it primary and deleted the old replica.

When I try this with datasets within this project, the replica creation process runs forever (I left it running for 5 days straight, with no result).

To workaround this, I tried to clone a dataset (which works fine) into a swap dataset, then delete the old one and clone the swap dataset into a new one with the same name as the original one. However, when I delete the dataset, it doesn't disappear from the dataset list. Only its tables are gone. When I click at it, it says "Not found: Dataset [...]". But when I try to create a dataset with the original name, I get "Already Exists: Dataset [...]".

So now I'm stuck. I cannot delete these datasets properly and I cannot create replicas for them. They seem to be broken somehow.

What can I do to fix them?

Thank you!

Cheers

Karsten189

The original issue was caused by the org policy "iam.allowedPolicyMemberDomains". Since the GCP project had been migrated into the organization, there were some users that had access to the dataset, but didn't belong to the organization / the allowed domains. Removing the access of these users from the datasets or adding them to the organization (in Cloud Identity) solved the issue, so that the replication processes finished successfully.

A special case is caused by search-console-data-export@system.gserviceaccount.com. You can find solution approaches to this one here: https://www.googlecloudcommunity.com/gc/Data-Analytics/Can-t-import-search-console-data-in-Big-Query...

View solution in original post