Solved: Troubleshooting Cloud Logging costs is so hard - Page 2

dheerajpanyam · 01-08-2025 08:38 PM

We seem to be incurring high Cloud Logging costs for one of our clients and their Infra spans a wide range of GCP services - GKE, Cloud Run Functions, Cloud Run, Firebase. It is hard to understand which of the logs are contributing to the costs. There are no destination sinks setup in our case

I totally understand the Cloud Logging architecture with exclusion and inclusion filters etc and also the GCP Audit log types but the challenges I am facing are mentioned below.

How do I figure out which log category my log message belongs and can I disable it or not for example for the application logs what category of log type does it belong to? Also I see some of the cloud run functions have messages that indicate Function started .. , Function ran for 110 seconds etc. What log types do these belong to (admin, data access, system event and can i disable them?
Another challenge is I might have enabled logs for some services , but there is no way to know upfront what those services are so the only way is to check log setting for each service which is cumbersome. A good example is VPC flow logs. VPC flow logs are disabled by default but you enable it , it generates additional log messages that will add to the costs.
Are there any custom log metrics / charts that i can setup to understand usage maybe a graph showing the volume of logs in each category (system logs, application logs etc)?

azzi

Dear dheerajpanyam,

I am 100% sure that the cause is the noise from too many logs written to the default bucket. The retaining period =< 30 days are included per this documentation. Even though you set the retaining period 20 days, the cost would still be the same https://cloud.google.com/stackdriver/pricing You only need to control how many GBs of logs ingested into a particular bucket.

As I previously stated, even though the GKE whether it is autopilot or standard stdout all logs but if your ingestion settings only include specific/granular parameter. You will not get charged for the non-ingested logs. You will only get charged for the logs that are included in the specific/granular parameter (except the Networking Telemetry I mentioned before ofc)

For the query, I'm not quite sure whether it is MQL or not. I think Google have it's own language. Please refer to https://cloud.google.com/logging/docs/view/logging-query-language

For the monitoring, is something like this sufficient for you?

To be able accurately track the ingestion, I suggest to add more filter to this monitoring such as filter by label. But unfortunately I don't see any way to label a Logging Bucket. Therefore I recommend you to create a new Cloud Storage Bucket then label it with appropriate key:value. After you finished creating the bucket, you then create a new sink with destination to that bucket that you just created.

Fyi, this is what I usually do. I disabled the default Sink, and created so many customs sinks so that I can control my logging cost more flexible. When I need something I just enable certain sinks, disable it again once I got what I need.

Hope this could give you some inspiration in your own GCP projects

Regards,
Iza

View solution in original post