unable to see "logs" or do "exec" in pod for autopilot private cluster

[shyam-uvik]

Hey All,

Cluster Architecture: private autopilot cluster with no external endpoint, and for connecting to api-server, using jump-host, which only is a way to connect to cluster and run kubectl commands. Below is the attached details of cluster and service account.



Issue: 
I have made a deployment of my application, but when i try to see logs or run "exec" command i get error saying :

Error from server: Get "https://10.250.82.36:10250/containerLogs/qa1/unity-app-5dd86f95dc-dmgjj/unity-app": dial tcp 10.250.82.36:10250: i/o timeout

earlier when i was trying to make deployment of [hasicorp-vault], i was getting same error, but after doing some R&D, i found out need to create serviceAccount and associate that to vault-deployment. than i was able to see logs or run "exec" command, but it used to run once for multiple tries. please look for vault-deployment screenshot for more context.

vault-deployment

so i tried same for mine application deployment, but i only am able to see error messages.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: unity-sa
  namespace: qa1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: unity-app
  namespace: qa1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: unity-app
  template:
    metadata:
      labels:
        app: unity-app
    spec:
      serviceAccountName: unity-sa
      containers:
        - name: unity-app
          image: app-image
          imagePullPolicy: Always
          resources:
            requests:
              memory: "500Mi"
              cpu: "750m"
            limits:
              memory: "900Mi"
              cpu: "990m"
          ports:
            - containerPort: 7100
          readinessProbe:
            httpGet:
              port: 7100
              scheme: HTTPS
              path: /unity/health
            initialDelaySeconds: 50
            periodSeconds: 40
            timeoutSeconds: 30
            failureThreshold: 5
            successThreshold: 1
          livenessProbe:
            httpGet:
              port: 7100
              scheme: HTTPS
              path: /unity/health
            initialDelaySeconds: 60
            periodSeconds: 30
            timeoutSeconds: 30
            failureThreshold: 5
            successThreshold: 1
          env:
            - name: MONGODB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: username

            - name: MONGODB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: password

            - name: VAULT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: vault-token

            - name: logger.path
              value: /var/log/unity
            - name: TZ
              value: "Asia/Kolkata"
          volumeMounts:
            - name: tz-config
              mountPath: /etc/localtime
              subPath: timezone
            - name: config-volume
              mountPath: /config
      volumes:
        - name: config-volume
          configMap:
            name: unity-app-config
        - name: tz-config
          configMap:
            name: timezone-config
---
apiVersion: v1
kind: Service
metadata:
  name: unity-app-cluster-service
  namespace: qa1
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
spec:
  selector:
    app: unity-app
  ports:
    - protocol: TCP
      port: 443
      targetPort: 7100
  type: ClusterIP
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: unity-role
  namespace: qa1
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [ "" ]
    resources: [ "pods/log" ]
    verbs: [ "get", "list", "watch" ]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: unity-role-binding
  namespace: qa1
subjects:
  - kind: ServiceAccount
    name: unity-sa
    namespace: qa1
roleRef:
  kind: Role
  name: unity-role
  apiGroup: rbac.authorization.k8s.io

i'm not able to understand what is the issue.

[nmagcalengjr]

Hi @shyam-uvik,

Welcome to Google Cloud Community!

It appears that a more thorough investigation of your project is necessary. For more detailed insights you may reach out to Google Cloud Support for assistance.