DataStream unable to start due to BigQuery connection profile

[g_mendes_gain]

I use pulumi as IaaS and yesterday we migrated from GCP's v6 to v7 and now the stream is unable to start.

This stream is running for almost 6 months syncing data from CloudSQL Postgres to Big Query and never had a problem.

When I try to start the stream I get the following:

---

The stream is connected to non-existent connectionProfile

---

I tried to debug it through Logs Explorer and there is no more information than that.

After the pulumi update, all the steps to create the stream have passed. Based on the GCP's operation cli response:

---

validationResult:
validations:
- code: POSTGRES_VALIDATE_CONNECTIVITY
description: Validates that Datastream can connect to the source Postgres database.
state: PASSED
- code: POSTGRES_VALIDATE_PUBLICATION
description: Validates that the publication exists and configured for the required
tables.
state: PASSED
- code: POSTGRES_VALIDATE_LOGICAL_DECODING
description: Validates that logical decoding is properly configured on the database.
state: PASSED
- code: POSTGRES_VALIDATE_REPLICATION_SLOT
description: Validates that the replication slot exists and not lost.
state: PASSED
- code: POSTGRES_VALIDATE_BACKFILL_PERMISSIONS
description: Validates that the given Postgres user has the select permissions
on the required customer tables
state: PASSED
- code: BIGQUERY_VALIDATE_API_ENABLED
description: Validates that the BigQuery API is enabled in the current project.
state: PASSED
- code: BIGQUERY_VALIDATE_DESTINATION_PERMISSIONS
description: Validates that Datastream has permissions to write data into BigQuery.
state: PASSED
- code: BIGQUERY_VALIDATE_DYNAMIC_DATASET_LOCATION
description: Validates that Datastream can write data into BigQuery datasets
in the specified location.
state: PASSED
verb: create

---

I checked the IAM policies and all the BigQuery-related permissions are granted to the DataStream service account. Reference: https://www.googlecloudcommunity.com/gc/Serverless/Unable-to-connect-datastream-to-a-bigquery-destin...

I also checked the KMS key for BQ encryption/decryption and the same assigned on the BigQuery dataset is assigned on the DataStream destination configuration.

How can I overcome this issue?

[Poala_Tenorio · Accepted solution]

It seems like the issue you're encountering is related to the connection profile for your Datastream stream not being recognized or available after migrating from GCP's v6 to v7. Here are some steps you can take to troubleshoot and potentially resolve the issue:

• Verify Connection Profile Configuration: Double-check the configuration of your connection profile for the Datastream stream. Ensure that it is correctly configured to connect to your CloudSQL Postgres database and BigQuery destination. Pay special attention to any changes that might have been introduced during the migration process.
• Recreate Connection Profile: If you suspect that the connection profile might have been corrupted or lost during the migration, try recreating it. Make sure to follow the correct steps and configuration parameters.
• Check Datastream Configuration: Review the configuration settings for your Datastream stream. Ensure that all necessary configurations, such as source and destination settings, are correctly set up. It's possible that there might be a misconfiguration causing the stream to fail to start.
• Review Logs: Although you mentioned that you didn't find much information in Logs Explorer, it's still worth reviewing all available logs for any potential error messages or clues as to what might be causing the issue. Look for any errors or warnings related to the stream startup process.
• Test Connectivity: Manually test the connectivity between your CloudSQL Postgres database and BigQuery destination using tools like psql or bq command-line utilities. This can help verify that there are no network or authentication issues preventing Datastream from connecting to either source or destination.