PosgreSQL to BigQuery Connection

[MacLopez]

I can't seem to connect the PostgreSQL source to BigQuery using Data Transfer Service and/or Data Stream

I already have the connection details as I have linked it directly to Looker Studio. However, it would be great if we also have it in BigQuery as possibilities are limitless. As mentioned, I already have the credentials (Username, Password, Host, Database name, Port) and the certificates and key (in .pem files). I only have the said credentials and files as the PosgreSQL source is managed by our affiliate.

Attempt 1. via Data Transfer Service

• I have tried filling out the information and the credentials but there is no way to upload the certificates. Which is why (I think) there's an error when trying to proceed or connect.

Attempt 2. via Data Stream

• I also tried creating a stream via Data Stream. Again, filled out the necessary information. We also created a connection profile where the credentials are needed but there's no option to upload the certificates?

I'm quite new to GCP and I also can't find a helpful step-by-step or how to on this topic. Please help.

[AlexET]

Hi @MacLopez,

Thanks for your question. This forum focuses on Google Cloud's Application Integration. Could you please confirm if the issue you're describing occurs while using Application Integration?

[MacLopez]

Hi @AlexET,

Thank you for the response. As I'm quite new to GCP, I'm not familiar with Application Integration. I'm using Big Query console if that helps with understanding my concern?
If this is not the correct forum, will you be able to point me to the appropriate one?
Or the application integration is something that we might consider when connecting to PosgreSQL?

[AlexET]

Hello! Absolutely, you can certainly take a look at this other forum - Data Analytics.

In the meantime, I'll reach out to our team to gather some insights on how Application Integration could potentially be a valuable option for your projects.

Please don't hesitate to reach out if you have any questions.

[PulkitMathur]

Hey @MacLopez , the choice of the tool depends on the amount of data you have to move and your use case.

For example, if you want to move a huge amount of data between systems, there are various tool like Dataflow etc in GCP’s data portfolio which might be a fit.

If your usecase is more in the realm of event-by-event processing, and you need to transform data, enrich data etc before it has to be ingested into the target, then Application Integration can be a fit. Feel free to take a look at Application Integration in our documentation 

[MacLopez]

Thank you both @AlexET @PulkitMathur,. I'll see if Application Integration fits what I need.   I'll check documentations in the meantime.