Custom Dashboards

[MarkAlexan]

I'm working with GCP Monitoring (Observability > Dashboards) and using a log-based custom metric created from logs ingested via Cloud Logging

What I am trying to do is to create a dashboard widget (Scorecard or Chart) that shows the total number of log entries (i.e., the total transactions) that match this metric over a given time range, I am using Alignment function "Delta".

The problem is when I increase the dashboard time range to 7 days or more, The chart or scorecard starts showing inaccurate totals.

How do I accurately count all emitted log-based metric points over a long duration?

[Joy_S]

Hi @MarkAlexan,

Welcome to Google Cloud Community!

To answer your question, the "Delta" alignment function in GCP Monitoring is designed to show you the change in your metric's value between consecutive alignment periods. While it's useful for understanding the rate of change, it's not the ideal function for calculating a cumulative total over a long duration. When you use "Delta" and then try to sum these changes over many hours or days, especially with the default longer alignment periods that GCP Monitoring applies for extended time ranges , small discrepancies or missed data points in each interval can accumulate and result in an inaccurate overall total.

You may want to check this post and try configuring your dashboard widget to use either the "Sum" or "Count" alignment function instead of "Delta." You might also want to consider adjusting the alignment period. For a 7-day view, the default alignment period might be 1 hour. You could experiment with a shorter period, such as 10 minutes, to see if it improves accuracy for your specific use case. 

You may also want to try to export logs to BigQuery for accurate counting potentially very long durations, and especially if you need to analyze historical data before your metric was created. You can set up a log sink in Cloud Logging to automatically export all your relevant logs to a BigQuery dataset and table. Once your logs are in BigQuery, you can use standard SQL queries to count the exact number of log entries that match your criteria over any time range you specify.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.