Solved: Re: [resolver] failed to query DNS server: 169.254...

mschewe · 10-31-2023 01:45 AM

Hello,

we are currently upgrading our nodes. After having updated some, we are regularly seeing the following error in our logs.

{
  "insertId": "1rtvd6vf99wb7e",
  "jsonPayload": {
    "daemon": "dockerd",
    "msg": "[resolver] failed to query DNS server: 169.254.169.254:53, query: ;swarm-node-01.c.yyyy.internal.\\tIN\\t AAAA",
    "error": "read udp 172.y.0.x:52672->169.254.169.254:53: i/o timeout",
    "level": "error"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "instance_id": "259009694028483yyyy",
      "zone": "europe-west1-b",
      "project_id": "yyyy"
    }
  },
  "timestamp": "2023-10-31T07:03:19.896244826Z",
  "labels": {
    "compute.googleapis.com/resource_name": "xxxx.c.yyyy.internal"
  },
  "logName": "projects/yyyyy/logs/syslog",
  "receiveTimestamp": "2023-10-31T07:03:20.735592166Z"
}

We are using the GCP in europe-west-1. We are running our own image based on the Google Cloud Debian 12 image. It's built with packer and has no configuration changes regarding networking.

This does not only occur when resolving internal resources, it happens with external domains too.

For me, this seems quite unusual, that a DNS server (aka the internal Google metadata server) gets an i/o timeout. Has anybody experienced some related issues?

RonEtch

Hi @mschewe

Welcome to Google Cloud Community!

These errors are produced during the provision of new nodes (either auto-provision or upgrade) since it creates a query to the compute engine metadata server. The said query is part of the compute engine API call that assures to resolve the new instance hostname to the internal DNS to be unique in the VPC network specially in zonal but not limited to global internal DNS.

Please note that metadata server requests to these servers (169.254.169.254) are capped per second to not to overwhelm one of the server's dependencies.

In summary, this can be non-concerning errors unless you're unable to update or upgrade your nodes.

You can learn more about internal DNS thru this guide.

I hope this information is helpful.

If you need further assistance, you can always file a ticket on our support team.

View solution in original post

RonEtch