Re: Cannot tail logs using DNS endpoint GKE

mehdi-relevanc · 04-24-2025 04:39 AM

Hi everyone,

I recently enabled the DNS endpoint to access the control plane of my GKE cluster. Everything's working smoothly, and I was able to remove the public IP from the control plane.

However, I'm running into an issue with streaming logs. When I run:

kubectl logs -f <mypod>

I get a blank output. But if I fetch logs without streaming, like this:

kubectl logs --since=1m <mypod>

it works just fine.

Has anyone else encountered this? Any ideas on what might be causing it?

Thanks!

mokit

Hi, @mehdi-relevanc.

Have you tried using this command (`kubectl logs -f <pod-name> --v=9`) to get more verbose log output and identify where it might be hanging or failing? If not, could you please try this? If it doesn’t reveal the root cause, feel free to share the output here to look further.

Regards,
Mokit

mehdi-relevanc

Here are the logs. When I perform this command nothing happens and it seems to hang

klf external-dns-5497678c79-c46g6 -v=9
I0425 09:06:13.222376   23799 loader.go:402] Config loaded from file:  /Users/mehdi.mahfoudi/.kube/config
I0425 09:06:13.222566   23799 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0425 09:06:13.222574   23799 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0425 09:06:13.222577   23799 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0425 09:06:13.222579   23799 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0425 09:06:13.225330   23799 helper.go:105] "Request Body" body=""
I0425 09:06:13.225370   23799 round_trippers.go:473] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.32.3 (darwin/arm64) kubernetes/32cc146" 'https://gke-xxxx.europe-west1.gke.goog/api/v1/namespaces/external-dns/pods/external-dns-5497678c79-c46g6'
I0425 09:06:13.235689   23799 round_trippers.go:502] HTTP Trace: DNS Lookup for gke-xxxx.europe-west1.gke.goog resolved to [{2001:4860:4802:32::1b } {216.239.32.27 }]
I0425 09:06:13.237622   23799 round_trippers.go:517] HTTP Trace: Dial to tcp:[2001:4860:4802:32::1b]:443 succeed
I0425 09:06:13.394419   23799 round_trippers.go:560] GET https://gke-xxxx.europe-west1.gke.goog/api/v1/namespaces/external-dns/pods/external-dns-5497678c79-c46g6 200 OK in 169 milliseconds
I0425 09:06:13.394475   23799 round_trippers.go:577] HTTP Statistics: DNSLookup 1 ms Dial 1 ms TLSHandshake 19 ms ServerProcessing 136 ms Duration 169 ms
I0425 09:06:13.394485   23799 round_trippers.go:584] Response Headers:
I0425 09:06:13.394496   23799 round_trippers.go:587]     Content-Length: 6590
I0425 09:06:13.394503   23799 round_trippers.go:587]     Audit-Id: 1e8499b1-7422-4d4f-aac2-103aac08c886
I0425 09:06:13.394507   23799 round_trippers.go:587]     Cache-Control: no-cache, private
I0425 09:06:13.394510   23799 round_trippers.go:587]     Date: Fri, 25 Apr 2025 07:06:13 GMT
I0425 09:06:13.394513   23799 round_trippers.go:587]     X-Content-Type-Options: nosniff
I0425 09:06:13.394516   23799 round_trippers.go:587]     X-Directpath-Agent-Latency: 25
I0425 09:06:13.394518   23799 round_trippers.go:587]     Content-Type: application/json
I0425 09:06:13.394533   23799 round_trippers.go:587]     X-Frame-Options: SAMEORIGIN
I0425 09:06:13.394536   23799 round_trippers.go:587]     X-Gkfe-Proxy-Success: true
I0425 09:06:13.394538   23799 round_trippers.go:587]     X-Kubernetes-Pf-Flowschema-Uid: 48ce012f-cbb0-444d-9e02-fbb92a55be19
I0425 09:06:13.394541   23799 round_trippers.go:587]     X-Kubernetes-Pf-Prioritylevel-Uid: c4863bdc-863c-416a-9d92-09481cad2b6b
I0425 09:06:13.394544   23799 round_trippers.go:587]     X-Xss-Protection: 0
I0425 09:06:13.394779   23799 helper.go:105] "Response Body" body=<
	{"kind":"Pod","apiVersion":"v1","metadata":{"name":"external-dns-5497678c79-c46g6","generateName":"external-dns-5497678c79-","namespace":"external-dns","uid":"39a5cf77-5ea7-41b2-861d-6b4bad2e8a1f","resourceVersion":"308901905","creationTimestamp":"2025-04-23T06:52:49Z","labels":{"app.kubernetes.io/instance":"external-dns","app.kubernetes.io/name":"external-dns","pod-template-hash":"5497678c79"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"external-dns-5497678c79","uid":"42ad3882-37c5-4b2c-a7ca-b670dcdc8913","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2025-04-23T06:52:49Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/name":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"42ad3882-37c5-4b2c-a7ca-b670dcdc8913\"}":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"external-dns\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":7979,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{},"f:capabilities":{".":{},"f:drop":{}},"f:privileged":{},"f:readOnlyRootFilesystem":{},"f:runAsGroup":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{".":{},"f:fsGroup":{},"f:runAsNonRoot":{},"f:seccompProfile":{".":{},"f:type":{}}},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2025-04-23T06:53:09Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PodReadyToStartContainers\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:hostIPs":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.1.2.11\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]},"spec":{"volumes":[{"name":"kube-api-access-rx59n","projected":{"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"name":"kube-root-ca.crt","items":[{"key":"ca.crt","path":"ca.crt"}]}},{"downwardAPI":{"items":[{"path":"namespace","fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}]}}],"defaultMode":420}}],"containers":[{"name":"external-dns","image":"registry.k8s.io/external-dns/external-dns:v0.16.1","args":["--log-level=info","--log-format=json","--interval=1m","--source=service","--source=ingress","--policy=upsert-only","--registry=txt","--provider=google"],"ports":[{"name":"http","containerPort":7979,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"kube-api-access-rx59n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":"http","scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":2},"readinessProbe":{"httpGet":{"path":"/healthz","port":"http","scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":6},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":65532,"runAsGroup":65532,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"external-dns","serviceAccount":"external-dns","automountServiceAccountToken":true,"nodeName":"gke-rc-infra-p-w7ld-k8s-euwe1-in-main-ac586e46-p4eo","securityContext":{"runAsNonRoot":true,"fsGroup":65534,"seccompProfile":{"type":"RuntimeDefault"}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"PodReadyToStartContainers","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:52:56Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:52:49Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:53:09Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:53:09Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-04-23T06:52:49Z"}],"hostIP":"192.168.192.91","hostIPs":[{"ip":"192.168.192.91"}],"podIP":"10.1.2.11","podIPs":[{"ip":"10.1.2.11"}],"startTime":"2025-04-23T06:52:49Z","containerStatuses":[{"name":"external-dns","state":{"running":{"startedAt":"2025-04-23T06:52:55Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"registry.k8s.io/external-dns/external-dns:v0.16.1","imageID":"registry.k8s.io/external-dns/external-dns@sha256:37d3a7a05c4638b8177382b80a627c223bd84a53c1a91be137245bd3cfdf9986","containerID":"containerd://cc9913993979b5373b52d38bde24e73edb993a29a438d200f0c47e028f047252","started":true,"volumeMounts":[{"name":"kube-api-access-rx59n","mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","readOnly":true,"recursiveReadOnly":"Disabled"}]}],"qosClass":"BestEffort"}}
 >
I0425 09:06:13.398564   23799 logs.go:469] "Request Body" body=""
I0425 09:06:13.398672   23799 round_trippers.go:473] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.32.3 (darwin/arm64) kubernetes/32cc146" 'https://gke-xxxx.europe-west1.gke.goog/api/v1/namespaces/external-dns/pods/external-dns-5497678c79-c46g6/log?container=external-dns&follow=true'

mehdi-relevanc

I did try with cloud shell and it works great, I believe it is because I have Netskope on my macbook.

garisingh

SO the issue was only on your Mac?

mehdi-relevanc

I believe so 🙃