Dataplane v2 - Cilium config api rate limits

[Moetazbellah]

recent versions of GKE > 1.25 , new rate limits applied in cilium-config 

api-rate-limit: '{"endpoint-create": "rate-limit:4/s,auto-adjust:false", "endpoint-delete":
"rate-limit:4/s,auto-adjust:false"}'
bpf-lb-service-backend-map-max: "262144"
bpf-lb-service-map-max: "262144"


is there any way to update that limits ?

[Moetazbellah]

it's noticed also in map list that throttling is enabled 

it's noticed also that cilium_throttle is enabled 

# cilium map list
Name                       Num entries   Num errors   Cache enabled
cilium_ipcache             262           0            true
cilium_lb4_backends_v3     152           0            true
cilium_lb4_affinity        0             0            false
cilium_lb4_services_v2     341           0            true
cilium_lb4_reverse_nat     520           0            true
cilium_lb_affinity_match   57            0            true
cilium_lb4_source_range    0             0            true
cilium_policy_00650        0             0            false
cilium_lxc                 7             0            true
cilium_policy_01064        0             0            false
cilium_policy_01259        0             0            false
cilium_localredirect       0             0            true
cilium_throttle            0             0            true
cilium_node_map            0             0            false
cilium_metrics             0             0            false

[garisingh]

We do not allow direct modifications to `cilium-config` as the defaults are currently set to recommended values suitable for most use cases.
It is possible to raise a support ticket and if support determines that the rate limit warrants a change, they can update the config.

[Moetazbellah]

yes, i opened a support ticket 
but for cilium and rate limits configurations , we need to leverage the powers of cilium and also have more flexibility
i'm posting here in case anyone with GKE dataplane v2 going from 1.25 to higher GKE , he might face similar hidden issues
 

[garisingh]

Sure - but do be aware that DPv2 is NOT the same thing as OSS Cilium, which is also why we do not expose all of the Cilium config options.

[Moetazbellah]

thanks for clarification 

do we have a separate release notes of DPv2 ? like the release notes of cilium ?
if we do have please share the link ?

for GKE release notes, i don't find much details around DPv2 issues and versions 
 i noticed the below by trial 

GKE 1.25. --> DPv2 (Cilium) 1.11.9
GKE 1.26 --> DPv2 (Cilium) 1.12.6
GKE 1.27 --> DPv2 (Cilium) 1.12.10
GKE 1.28.5 --> DPv2 (Cilium) 1.13.9

but i couldn't find the release notes of DPv2 1.12.10 ? and couldn't find the reason behind adding rate limits in DPv2 releases > 1.11.9

we faced different issues from one release to another like out of order events 

we opened a support case with GCP but couldn't reach to resolution and also couldn't find relevant documentation for each release where i can define the root cause of an issue 

[garisingh]

We don't have separate release notes for DPv2.  Just the standard GKE release notes for everything.   But it's a fair ask.

https://issuetracker.google.com/issues/327814630 is the bug you filed, correct?

[Moetazbellah]

Correct

[Rouf]

I'm running GKE 1.28  and experiencing the issues with the default API rate set in Cilium config. 

api-rate-limit: '{"endpoint-create": "rate-limit:4/s,auto-adjust:false", "endpoint-delete":
"rate-limit:4/s,auto-adjust:false"}'
ISSUE:
plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequest

I have attempted to adjust these settings, but my changes are getting reverted. Are there plans to allow users to modify these options if they encounter problems? What alternative solutions do you suggest to address this issue?

[alextricity25]

I'm also getting the same thing. 

I'm running vcluster on GKE 1.30.3-gke.1451000, which may have something to do with why I am running into these limits in the first place. I would really like to see a way for users to adjust these settings in a persistent fashion.

These are the errors I'm seeing:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5c81c1a4b8b9f6783275d2fdb12659a2d9b4476736636be631c059f25d62dc1c": plugin type="cilium-cni" failed (add │
│ 😞 unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9a68469f09d79d1dc466ae5ff8c3fc005bb5af674ad302b1f363a0a006dcdaf5": plugin type="cilium-cni" failed (add │
│ 😞 unable to create endpoint: Cilium API client timeout exceeded

@Rouf Have you had any luck with support or have you found anything interesting in the documentation?

[minhnnhat]

Same for me with GKE 1.32.1-gke.1357001. With the error:
```
Warning FailedCreatePodSandBox 48s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f1b0dd4ef19dd183c91bd9674cb6628a3b82c1b1ab8c7e5f05fc6bae067460e7": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests
```