Connect Dataform with Bitbucket Server

[danielopezNext]

Hello,

I need to connect Dataform with the local Bitbucket repository.

My question is if I can connect 'DataForm' repositories with Bitbucket Server.

Is there any way to make this need possible?

I have seen in the documentation that it is possible with Bitbucket Cloud:

https://cloud.google.com/dataform/docs/connect-repository#authenticate_a_remote_repository_through_s...

and I have also seen that it is possible to connect Bitbucket Server with Cloud Build.

https://cloud.google.com/build/docs/automating-builds/bitbucket/connect-repo-bitbucket-server

https://cloud.google.com/build/docs/automating-builds/bitbucket/build-repos-from-bitbucket-server-pr...

Thanks,

regards.

[ms4446 · Accepted solution]

Currently, it is not possible to connect Dataform repositories with Bitbucket Server. In the meantime, here are a few potential workarounds that you might want to consider:

1. Using a Third-Party Git Repository Hosting Service:

   • Clone your Bitbucket Server repository to GitHub or GitLab.
   • Connect your Dataform repository to the GitHub or GitLab repository.
   • Ensure synchronization between your Bitbucket Server repository and GitHub or GitLab so that changes are mirrored.
   • Dataform will automatically build and deploy your repository when changes are pushed to GitHub or GitLab.

2. Using a Cloud Build Trigger:

   • Configure a webhook on your Bitbucket Server repository to notify Cloud Build when changes are pushed.
   • Create a Cloud Build trigger that listens for events from the webhook.
   • Configure the Cloud Build trigger to build your Dataform repository.
   • Cloud Build will trigger a build of your Dataform repository when changes are pushed to your Bitbucket Server repository.

3. Using a Custom CI/CD Pipeline:

   • Create a custom CI/CD pipeline that builds and deploys your Dataform repository.
   • Configure the CI/CD pipeline to listen for events from your Bitbucket Server repository.
   • The CI/CD pipeline will be triggered, and your Dataform repository will be built and deployed when changes are pushed to your Bitbucket Server repository.

These are potential workarounds, and their effectiveness may depend on your specific setup and requirements. It's always a good practice to test and verify each solution in a controlled environment before deploying it to production.

[tushensu]

Hi is this still the case ? I am working on a Dataform solution and in the company I recently joined the main source tool is Bitbucket. I have tried connecting to bitbucket using the HTTPS method and it does not work with the access token. 

[danielopezNext]

@tushensu 

The solution was through ssh, and my case is with the bitbucket server.

But I'll explain the connection if it helps you.

You must create a secret manager with the value of the bot or user private key to connect to bitbucket.

-----BEGIN OPENSSH PRIVATE KEY----- y -----END OPENSSH PRIVATE KEY-----

Then set bitbucket access to 'Writer' in the bitbucket repository.

Finally in the Dataform configuration add this SecretManager and the ssh-rsa key that you use on the server to download the repository.

[tushensu]

I am not sure what I'm doing wrong here but, this is what I have done .

1. Generated SSH Keys using open ssh on a windows machine

2. Added the public key onto my bitbucket personal settings (I also tried adding to workspace settings)

3. Created a Secret with the private key in GCP secrets manager

4. In Dataform, added my ssh url for bitbucket repo, chose the secret I just added, pasted the public key I added onto Bitbucket into the host public key section of the Dataform integration.

I am still getting this error below

[danielopezNext]

That's see good,
the only one thing i think,
if you have set the user permissions in bitbucket repository configuration to 'Writer'.
The user must be the same of the secret manager key.

And other thing be carefull with url ssh, must be the same hen you clone tthe repository.

[tushensu]

---

"if you have set the user permissions in bitbucket repository configuration to 'Writer'.
The user must be the same of the secret manager key"

When you say the user must be the same what do you mean ? When I create the keys, is there a user I can explicitly say they belong to ? 

 Like is there something from GCP that has username details that needs to match with Bitbucket ?

[tushensu]

@danielopezNext  I finally figured out what I was doing wrong, on the dataform git connection configuration page, I was suppose to add the bitbucket host public key which is available on bitbucket and not my public key (which is what I was adding).

This solved my problem. Thank you for the help and guiding me through this 👍 

[sri919]

@tushensu Where did you get the bitbucket host public key from? Thanks