My Persistent Disk SSD quota is almost exceeded, despite the fact that I did not fill anything

[Bruce_Q]

Hi!
Could you please help me

Context:
- I created a new project
- I created a auto-pilot GKE cluster
- I launched a super lightweight pod in the cluster
- I launched a gitlab runner in the cluster

Problem:
However, new pods are no longer created - because of the "0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod" error

In IA&Admin -> Quotas&System Limits i see that "My Persistent Disk SSD" ("service" column value = Compute Engine API) quota is almost exceeded, 200GB/250GB is already in use, despite the fact that I did not fill anything

Debugging:

gcloud compute disks list -> empty

kubectl top nodes -> 

NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
<pool-1> 169m 8% 1927Mi 32%
<pool-2> 241m 12% 2508Mi 42%

kubectl describe node <pool-1> ->

Resource Requests Limits
-------- -------- ------
ephemeral-storage 1Gi (2%) 1Gi (2%)

kubectl describe node <pool-2> ->

Resource Requests Limits
-------- -------- ------
ephemeral-storage 0 (0%) 0 (0%)

I'm not new in k8s, but I'm using GCP for the first time

Please tell me how to see what's taking up so much space?

[reinc]

Hi @Bruce_Q, 

Welcome to Google Cloud Community! 

This error “0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod” occurs because Autopilot is still provisioning the underlying nodes required to run your Pods. This process can take a while, especially in a new cluster.

The fastest way to get your cluster functioning again is to Request a Quota Increase for "Persistent Disk SSD" in the Compute Engine API. This will give you more headroom and allow Autopilot to provision new nodes.

1. Go to Google Cloud Console > IAM & Admin> Quotas
2. Filter the following:

• Service: "Compute Engine API"
• Metric: "Persistent Disk SSD (GB)"

3. Click "Edit Quotas" at the top.
4. Enter the desired new quota value (ex. try increasing from 250GB to 500GB). Explain your reason for requesting this increase (ex. "GKE Autopilot cluster requires more disk space for node provisioning").
   

Free Tier accounts include a limited amount of Persistent Disk SSD storage each month. You can find details about the Google Kubernetes Engine Free Tier. For a related issue, see this Google Cloud Community Question. 

If you need further assistance and any questions, please reach out to our Google Cloud Support team.


Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.