Create dashboard from cloud logs

[shikhar18]

i have some dataproc batch (serverless) jobs which read data from gcs and write to bigquery. In my job i am capturing some counts using df.count() and then finally writing this count as json logs using from google.cloud import logging , logging.log_struct({"count":df.count()}) . I can see the count in a json format in the cloud logging but when i am trying to create a metric from this log i am only seeing counter or distribution type metric.

I want to create a dashboard where it just plots the count value for each run . Is there any other way to achive this ? or i need to go with creation of distribution type metric and then creating a dashbaord from this metric in Monitoring ?

[ms4446]

You can create a dashboard in Cloud Monitoring that plots the count value for each run of your Dataproc batch jobs. Instead of using a distribution type metric, you can utilize logs-based metrics.

Steps to Create a Logs-based Metric from Cloud Logging:

1. Log Filter Creation:

   • Go to the Cloud Logging console in the Google Cloud Console.
   • In the left-hand navigation pane, click on Log Explorer.
   • Create a filter to capture the relevant logs. For example:
     resource.type="global" AND jsonPayload.count IS NOT NULL
   • Ensure that the resource type and log fields match your specific logging setup.

2. Create a Logs-based Metric:

   • From the results of your filter, click on "Create Metric".
   • Name your metric and set the metric type to "Counter".
   • In the "Field" section, use the JSON path to extract the count from your log entry. If your log entry looks like {"count": 123}, the JSON path would be $.count.

Steps to Create a Dashboard in Cloud Monitoring:

1. Navigate to the Cloud Monitoring console.
2. In the left-hand navigation pane, click on Dashboards.
3. Click the Create Dashboard button.
4. Name your dashboard.
5. Click the Add Chart button.
6. In the Resource type and metric section, select the logs-based metric you created earlier.
7. Configure the chart as needed and click Save.

Your dashboard will now display the count value for each run of your Dataproc batch jobs.

Note: While the above method directly plots the count values, you also have the option to use distribution type metrics for a more detailed distribution view of the count values. For more detailed steps and options, refer to the official Google Cloud Monitoring documentation.

[shikhar18]

@ms4446  : When i try to create a log based metric and select metric type as counter it does not give any option "Field" section rather it only gives option only for Build Filter. 

As per my understanding the counter metric will just count number of times a log entry event happens based on the the filter we pass in Build Filter .

[ms4446]

You are correct. The "Field" section is only available for distribution type metrics. For counter type metrics, the metric value is simply the number of log entries that match the filter you specify.

In the example you provided, the filter resource.type="global" AND jsonPayload.count IS NOT NULL will count the number of log entries that have a resource type of "global" and a non-null value in the "count" field.

Once you have created the counter type metric, you can add it to a dashboard in Cloud Monitoring to visualize the count value for each run of your Dataproc batch jobs.

Please note that the steps for creating a logs-based metric may vary depending on the version of Cloud Logging you are using. For more information, please see the Cloud Logging documentation: https://cloud.google.com/logging/docs.

[shikhar18]

Once i have created the counter metric with the specified filters. How will the value from jsonPayload.count be plotted in the Monitoring dashboard ?
As per my understanding in monitoring dashboard will just be able to plot number of times that log entry has come and not the value inside jsonPayload.count. 

[ms4446]

You're correct in your understanding. When you create a counter metric in Cloud Logging, it counts the number of log entries that match the specified filter. It does not capture or plot the actual values inside a specific field like jsonPayload.count.

If you create a counter metric with the filter resource.type="global" AND jsonPayload.count IS NOT NULL, the metric will increment each time a log entry matches this filter. In Cloud Monitoring, the dashboard will show the number of times this log entry occurred over a specified time period, but it will not show the actual values inside jsonPayload.count.

If you want to capture and visualize the actual values inside jsonPayload.count, you would need to use a distribution metric. A distribution metric allows you to create histograms based on the values in specific fields of your log entries. This way, you can see the distribution of values inside jsonPayload.count over a specified time period.