CloudBuild Private Pool unable to reach Cloud DNS

[digantakoner]

Hello Everyone,

We have a situation wherein the CloudBuild private pool is not able to reach the Private Cloud DNS Zone.

Description of the environment-

A cloudbuild private worker pool is configured to run builds as per triggers configured in cloudbuild. The worker pool has a ip range of 10.193.164.0/22. The subnet is connected to a VPC network using Private Service Connection. A VM is hosted on the same VPC network which exposes couple of APIs. There is a private DNS zone setup in Cloud DNS which provides the DNS resolution for the network.

Issue-

CloudBuild private pool is able to access the API exposed by the VM using the ip address of the VM. However, when the hostname is used, the DNS resolution fails and cloudbuild worker pool is unable to access the VM.

Ask-

Is there a way to configure Cloud Build private pool so that it can leverage Cloud DNS to resolve the VM name?

Thank You,

Diganta Koner

[michael_li · Accepted solution]

@digantakoner Thank you for the information. I have tested it with the following terraform code example, and it works. The private cloudBiuld worker can use my private DNS zone and is able to connect to the VMs in my VPC network.

## Uncomment this block after adding a valid DNS suffix

# resource "google_service_networking_peered_dns_domain" "default" {
#   name       = "example-com"
#   network    = google_compute_network.peering_network.name
#   dns_suffix = "example.com."
#   service    = "servicenetworking.googleapis.com"
# }

[Marvin_Lucero]

Hi @digantakoner ,

Your question may have been lacking information such as the error message when the VM is accessed via hostname, or if there were guides or documentations that you followed prior to this setup. 

---

@digantakoner wrote:

CloudBuild private pool is able to access the API exposed by the VM using the ip address of the VM. However, when the hostname is used, the DNS resolution fails and cloudbuild worker pool is unable to access the VM.

---

If this was the case, here are some potential causes and solutions that you can consider:

1. Ensure that the VM's DNS configuration is accurate by confirming the DNS settings within the VM. Validate that the hostname is appropriately registered and linked to the correct IP address. Additionally, verify that the DNS server employed by the CloudBuild private pool has the essential DNS records for the VM's hostname. 

2. Check the DNS resolution settings within the network or VPC where the CloudBuild private pool is running. Ensure that the DNS resolver used by the pool is correctly configured to query the DNS server that contains the hostname-to-IP mapping for the VM.

3. Check for any network restrictions, firewall rules, or security groups that may be blocking DNS queries from the CloudBuild private pool workers. Ensure that the necessary ports and protocols for DNS resolution are allowed.

---

@digantakoner wrote:

Is there a way to configure Cloud Build private pool so that it can leverage Cloud DNS to resolve the VM name?

---

Regarding to your question, this documentation about running builds in private pool might help you.

[digantakoner]

Hey Marvin,

Apologies for leaving the documentation and error out.

Documentation followed - 

Running builds in a private pool  |  Cloud Build Documentation  |  Google Cloud

https://cloud.google.com/dns/docs/best-practices#reference_architectures_for_hybrid_dns

however, I did not find anything about setting DNS for cloudbuild worker pools. If there is something I am missing please provide me the documentation or URL.

The DNS of the VM is working fine and is able to connect from other VM in the same VPC.

Error - When the cloudbuild worker tries to reach the VM using the DNS name, the connection times out and the timeout error pops up.

[michael_li]

@digantakoner I have the same requirement which needs the cloud build worker using GCP private DNS zone from the google cloud build private worker pool. Unfortunately, it seems the Google Cloud build private worker pool can use public DNS servers only. 

Since we can't configure DNS peering between the GCP producer network and your private network, The cloud build worker won't be able to use your private DNS zone.

A workaround solution is to add a host entry to the /etc/hosts file into the cloud build worker that you can specify a command line, "echo ${private-ip-of-your-gcp-vm} your-domain-name",  in the build steps which you need to reach your GCP VM.

Please note you need to modify the /etc/hosts file in each of the steps which you need to reach your GCP VM because it refuses to modify the /etc/hosts file permanently between the cloud build steps. We can't save the modified /etc/hosts file, nor the /etc/resolv.conf file as a volume and pass it to other steps.

I am also investigating if there is a better way to resolve the issue. If you find one, kindly share your solution. Many thanks.

[michael_li]

@digantakoner I think I found the solution. We can configure DNS peering between the user-managed VPC network and the Google producer network. We can use Terraform to add the DNS peering. Here is the terraform resource, https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_service_networ....
I haven't tested it, and I will try it tomorrow.
Here are some other reference links,
https://cloud.google.com/build/docs/private-pools/use-in-private-network#dns-zones
https://cloud.google.com/vpc/docs/configure-private-services-access#terraform_2

[digantakoner]

@michael_li Thank you so much for the approach you provided. The terraform approach is something I did try, however the issue is it still reverts to an external service provider, something that you mentioned in your previous reply. Since cloudbuild on a private network is peered to the VPC, DNS through Private Service Connection doesn't seem to work(atleast that's what the Google docs say). 

If this approach or any other solution has worked for you, kindly post it. Would love to follow that. 

[michael_li · Accepted solution]

@digantakoner Thank you for the information. I have tested it with the following terraform code example, and it works. The private cloudBiuld worker can use my private DNS zone and is able to connect to the VMs in my VPC network.

## Uncomment this block after adding a valid DNS suffix

# resource "google_service_networking_peered_dns_domain" "default" {
#   name       = "example-com"
#   network    = google_compute_network.peering_network.name
#   dns_suffix = "example.com."
#   service    = "servicenetworking.googleapis.com"
# }

[michael_li · Accepted solution]

Here are the cloud build trigger steps I tested for the connection.

steps:
  - name: gcr.io/cloud-builders/git
    args:
      - '-c'
      - |
        apt-get update
        apt-get install dnsutils -y
        echo "dig test.example.com ..."
        dig test.example.com
        curl test.example.com
    id: check the private DNS zone and the private connection
    entrypoint: bash
options:
  workerPool: >-
    projects/myporject/locations/us-east4/workerPools/myproject-mynetwork-cloud-build-worker-pool

[digantakoner]

@michael_li This worked like a charm. You saved my day. Thank You so much for the solution. Have been scratching my head for quite sometime now, but guess didn't look at the right place. Cheers mate!

[michael_li]

@digantakoner Thank you for the update. I am so glad to hear it worked on your end too. Cheers mate!