Lack of Cost Visibility and Resource Optimization in a Large Shared GKE Cluster

[jsonlynch]

As our shared GKE cluster scales to accommodate a growing number of applications, we face a critical challenge: the inability to track and charge back costs per application or namespace. Currently, I'm finding it hard to look for the granular cost visibility at the namespace or application level, making it difficult to enforce accountability and optimize resource usage.

• Unaccounted Costs – No visibility into per-app/namespace spending.
• Resource Misuse – Developers copy-paste CPU/memory requests without analyzing actual usage.
• Scaling Inefficiencies – Overprovisioning increases costs and impacts scheduling.

To ensure cost transparency, efficient resource use, and governance, we need a solution that provides cost breakdowns, optimizes resource allocation, and prevents misuse to use the node resources efficiently. 
If you have any advice or solves this problem, please do share.

[jayeshmahajan · Accepted solution]

I'm assuming you have used label for each namespace and its challanging to use GCP Cost Management (Billing Reports, Cost Allocation Tags, BigQuery Export) to track costs at the namespace and application level then my suggestion would be to try goldilocks that will give you pod/resource level recommendations based on cpu and memory usage by looking at prometheus metrics. Yes, you will need Prometheus or some other monitoring tools that can give that metrics for cpu and memory to pull the information for actual usage. 
Keda is another one to optimize the unnecessary workload as long as you know how to manage application and its availability.

[jayeshmahajan · Accepted solution]

I'm assuming you have used label for each namespace and its challanging to use GCP Cost Management (Billing Reports, Cost Allocation Tags, BigQuery Export) to track costs at the namespace and application level then my suggestion would be to try goldilocks that will give you pod/resource level recommendations based on cpu and memory usage by looking at prometheus metrics. Yes, you will need Prometheus or some other monitoring tools that can give that metrics for cpu and memory to pull the information for actual usage. 
Keda is another one to optimize the unnecessary workload as long as you know how to manage application and its availability.

[mokit · Accepted solution]

Hi, @jsonlynch.

As far as I know, GKE doesn't provide native cost breakdowns at the namespace or resource-use level. It only offers cost breakdowns at the cluster, region or tag level.

However, there are third-party tools like Kubecost which have become quite popular for exploring more granular cluster-specific cost breakdowns. You might want to give it a try.

Regards,
Mokit