Dataform - Unable to connect to your Git provider

[rafael-bg]

We are trying to connect the Repos (Azure DevOps) as the main repository of Datafrom (GCP) and we are encountering the following error.

We are following exactly what the documentation requires:

1. Generating an SSH key pair linked to the service account used in Dataform (ssh-keygen rsa -c "email@sa-dataform")
2. Placing the private key in a secret in the Secret Manager
3. Adding the public key to the repository (Azure DevOps)
4. Configuring the repository in Dataform

Show More

We are unable to connect to your Git provider with the configured credentials. If you use HTTPS authentication check that the token is valid and has not expired. If you use SSH authentication check that the private user key and public host key are valid. View docs on connecting a remote git repository. 

To make matters worse, when trying to connect to the external repository and even after deleting the configuration attempt, the process broke my version release plan.

Somebody help us?

[ms4446]

This error is quite common, but often stems from a few key areas. Here are some troubleshooting steps you can take:

• SSH Key Pair Verification:

  • Validity: Ensure your SSH key pair is correctly generated (RSA, minimum of 2048 bits). Verify the bit length and type with the command:

    ssh-keygen -l -f /path_to_your_public_key 
    

    If your key is not in OpenSSH format, convert it using:

    ssh-keygen -p -m PEM -f your_private_key 
    

  • Correct Key Upload: Ensure the public SSH key is correctly added to your Azure DevOps user settings under SSH keys. The private key should be securely stored in Google Cloud Secret Manager.

  • SSH Test: Test the SSH connection directly from your environment using:

    ssh -i /path/to/your/private_key git@ssh.dev.azure.com 
    

    Replace /path/to/your/private_key with the actual path to confirm if the SSH connection is working.

• Secret Manager Configuration:

  • Secret Name: Double-check the secret's name in your Dataform repository configuration for typos. It should exactly match the one used in Dataform.
  • Secret Version: If multiple versions of a secret exist, ensure Dataform is configured to use the correct, active version containing your private SSH key.
  • Access Control: Confirm that the Dataform service account has the Secret Manager Secret Accessor role on the secret containing the private key.

• Network Connectivity:

  • Firewall Rules: Check that firewalls between your Dataform environment and Azure DevOps allow outbound traffic on port 22 (SSH). Test connectivity with:

    nc -zv ssh.dev.azure.com 22 
    

    or

    ssh -T git@ssh.dev.azure.com 
    

  • Proxy Settings: Ensure correct proxy settings are applied to the Dataform service account and that the proxy permits SSH connections.

  • SSH Config File: For complex network setups, consider creating an SSH config file (~/.ssh/config) with specific proxy settings or host aliases.

• Dataform Configuration:

  • Repository URL: Verify that the repository URL is correct and points to the specific Azure DevOps project and repository, typically formatted as:

    git@ssh.dev.azure.com:v3/organization/project/repository 
    

  • Authentication Type: Confirm that "SSH" is selected as the authentication type in Dataform's repository settings.

  • Test Connection: Utilize the "Test Connection" feature in Dataform to verify if the setup is correct.

• Additional Considerations:

  • User Permissions: Ensure the user configuring the repository in Dataform has necessary permissions (read/write/execute) in both Azure DevOps and Google Cloud.
  • Dataform Service Account: Verify that the Dataform service account has the appropriate roles, including Secret Manager Secret Accessor and the necessary Git permissions in Azure DevOps.