GKE Gateway not ready - missing networkEndpointGroup

[jamesk-prefab]

My Gateway (Yaml Below) is stuck at Pending due to a resource not found with this error message

Conditions:
Last Transition Time: 2023-02-13T22:28:38Z
Message: error cause: gceSync: generic::not_found: Update: The resource 'projects/pfab-cloud-staging/zones/us-central1-f/networkEndpointGroups/k8s1-86239e94-kube-system-gw-serve404-80-b197d7d7' was not found
Reason: Pending
Status: False
Type: Ready
Name: https

I'm not sure how to debug this - I'm just testing so i removed the gateway, httproutes and services and waited a bit before putting it back again. This definition previously worked before i was trying to update the TLS termination settings (not allowed with a cert-map).

Looking at network endpoint groups i see two endpoints kube-system-gw-serve ending in b197d7d7 but only in zones us-central1a and us-central1c, while this is looking for us-central1f and us-central1b

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: external-http
  namespace: prefab
  annotations:
    networking.gke.io/certmap: prefab-cert-map
spec:
  gatewayClassName: gke-l7-gxlb
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        kinds:
          - kind: HTTPRoute
    - name: https
      protocol: HTTPS
      port: 443
      allowedRoutes:
        kinds:
          - kind: HTTPRoute
  addresses:
    - type: "NamedAddress"
      value: "${data.google_compute_global_address.global_public_ip.name}"

[Marvin_Lucero]

Hi @jamesk-prefab ,

Based from your error, there were reported issues similar to what you're encountering saying "networkEndpointGroups...not found". Upgrading to GKE master version 1.25 and higher should fix this issue. You can follow this guide for more details.