In Dataplex, you cannot directly scan or manage BigQuery assets located in a different GCP organization. Dataplex enforces organizational boundaries for security and isolation, which means it does not support direct access to BigQuery datasets across different organizations. This is why you encountered the error stating that the referenced resource is not in the same organization as the asset.
Google Cloud enforces strict boundaries between organizations to maintain security and isolation of resources. This restricts Dataplex from directly accessing BigQuery datasets in another organization. Dataplex relies on IAM for access control. Permissions granted to Dataplex service accounts are scoped within the organization where the account was created. Dataplex cannot authenticate or authorize access to BigQuery datasets in another organization using its service account.
Alternative Solutions
1. Use BigQuery Views:
- Create a View in Org A: You can create a BigQuery view in Org A that references the dataset in Org B. This view acts as a proxy to the dataset in Org B.
- Grant Permissions: Ensure that the Dataplex service account in Org A has the necessary permissions (
BigQuery Data Viewer) on the dataset in Org B to access the view.
- Add the View to Dataplex: Add this view as an asset in Dataplex in Org A, allowing Dataplex to scan and manage it.
2. Use BigQuery Data Transfer Service:
- Set Up Data Transfer: Use the BigQuery Data Transfer Service to regularly copy data from Org B’s dataset to a dataset in Org A.
- Add the Transferred Dataset to Dataplex: Manage this dataset in Dataplex as a local asset within Org A.
3. Data Replication via ETL Tools:
- Replicate Data: Use ETL tools like Dataflow to replicate data from BigQuery in Org B to BigQuery in Org A.
- Manage in Org A: Once replicated, add the dataset to Dataplex in Org A.
- Views in Org A act as a representation of data from Org B, allowing Dataplex to interact with the view without direct access to Org B’s dataset.
- These methods bring the data into Org A, eliminating cross-organization access restrictions.
If you need to scan a dataset orgb_project.sales_data in Org B using Dataplex in Org A:
- Create a View in Org A:
CREATE OR REPLACE VIEW `orga_project.sales_view.sales_data` AS
SELECT * FROM `orgb_project.sales_data.table_name`;
- Grant Permissions in Org B: Grant
BigQuery Data Viewer to the Dataplex service account from Org A on the dataset in Org B.
- Add the View in Dataplex: Add
orga_project.sales_view.sales_data as an asset in Dataplex in Org A.
while Dataplex cannot directly access BigQuery assets in a different organization, you can use views or data transfer methods to work around this limitation by effectively bringing the data into the same organization as your Dataplex setup.
LinkedIn
Twitter
