Unable to connect datastream to a bigquery destination in another project

[rs_cwc2023]

The plan is to create a MySQL(projectA)-Datastream(project-B)-BigQuery(project-C) stream.

I could create a source connection from projectB to projectA. However, I am unable to create a destination connection from project-B to project-C.

The default Datastream service agent(projectB) has these permissions in project C - BigQuery Editor, viewer and job user. Whenever I click validate the stream - it is creating the the dataset in the destination but I am getting this error - "The requested bigquery dataset doesn't exist, or Dtaastream does not have permissions to view it"

Are there any more permissions that I need to give to the service agent in project C or does it require any other network configuration changes? Thanks in advance!

@ms4446 

[ms4446]

There are several potential issues:

1. The destination BigQuery dataset does not exist.
2. The Datastream service agent does not have access to the destination BigQuery dataset.
3. The Datastream service agent cannot access the destination BigQuery dataset over the network.

Here's a step-by-step guide to ensure you've set up the necessary permissions:

1. Check that the destination BigQuery dataset exists.
2. Check the dataset's IAM policy to ensure that the Datastream service agent has the following roles:
   • BigQuery Data Editor
   • BigQuery Dataset Viewer
   • BigQuery Job User
3. Try creating a new BigQuery dataset in project C and granting the Datastream service agent access to it.
4. Try re-configuring the Datastream connection or setting up a new connection from MySQL (project A) to BigQuery (project C).
5. If you are still having problems, please provide more information about your Datastream configuration or version. You can also provide the full error message that you are receiving.

Additional Notes:

• Datastream is a fully managed service, so it should ideally handle network connectivity without requiring any advanced networking configurations.
• If the destination BigQuery dataset is in a different network than the Datastream service agent, you may need to configure network connectivity between the two networks using a VPC peering or Cloud Interconnect connection.
• If the destination BigQuery dataset is in the same network as the Datastream service agent, but the service agent cannot access the dataset, you may need to troubleshoot the network connection. This could involve checking firewall rules or any other network configurations that might be blocking the connection.