Using Dataform with Internal/Private GitLab Repositories

[Guillaume67]

Hello Google Cloud Community,

I am looking to use Dataform to execute data pipelines in my company. We use GitLab for version control, but due to security restrictions, we can only create internal or private repositories, which is currently incompatible with Dataform.

I would like to know if there is a way to version Dataform projects without using GitLab, or if a local git would be a viable option.

Thank you in advance for your help!

Best regards,
Guillaume

[caloric8585]

Yes. Dataform within GCP has it’s own interface that allow your team members to craft a commit straight from the browser and push to main. Make sure to familiarise yourself with the Dataform’s concept of ‘workspace’.

I’m confused. Docs mention GitLab support in first sentence and it’s obvious private repos should be fine: https://cloud.google.com/dataform/docs/connect-repository . I guess more important is whether your team already use BigQuery and what you are trying to do.

P.S. I sense I’m responding to chat bot:)

[mars124]

I can confirm that private GitLab and GitHub repositories can be connected to BigQuery's Dataform. My Dataform repositories are linked to private repositories. I'm not sure what you meant by "currently incompatible".

[Guillaume67]

Hi,

Thank you for your response! I suspect that the issue might be due to the fact that I am behind my company's firewall and the repository is not accessible to the public internet.

Is this the case for you as well?

Guillaume

[danielholgate]

Hi Guillaume,

In the documentation for connecting to repositories https://cloud.google.com/dataform/docs/connect-repository
there is a note about connecting to repositories behind firewalls. You need to fill in this form https://cloud.google.com/dataform/docs/connect-repository and will then be given fixed egress IP addresses which you can open your fire wall to and then Dataform should work for you.

regards

Daniel

[Guillaume67]

Hello Daniel,

I have submitted the request by filling out the form mentioned in the documentation for connecting repositories behind a firewall.

Could you please let me know when we can expect to receive these details?

Thank you in advance for your help!

Best regards,
Guillaume

[danielholgate]

Hi Guillaume,

it generally takes a week or two to get the IP addresses allocated.

regards

Daniel