LinkedIn
Twitter
We are continuing our mini-series around dashboards in Google Security Operations (SecOps) today with a look at grouping data in a chart. Data grouping provides us the ability to visualize data with additional dimensionality. Oftentimes, this dimension is time, but it doesn’t have to be, though it will be today. While we are at it, we’ll use the detection dataset so you can expand your charts beyond the UDM data we previously discussed.
Generally, when we talk about grouping data, we’re talking about aggregation. When generating a count of the number of events like we did last time, we aggregated or grouped our events by the metadata.event_type field and then counted them. To do this in our search or chart, we define the field or variable that the group will occur on in the match section of our search.
metadata.event_type = $event_type
match:
$event_type
outcome:
$event_count = count(metadata.event_type)
order:
$event_count desc
Today, we are going to do the same thing but then add time in as another dimension, which we will do incrementally. In our dashboard, we can click Add to create this new chart. The initial search is going to follow the same general layout as our previous one except we will be using the detections dataset.
$severity = detection.detection.severity
detection.detection.rule_set != ""
detection.detection.alert_state = "ALERTING"
match:
$severity
outcome:
$severity_count = count($severity)
Our initial query is looking for curated detections that generate alerts. We are grouping by the severity of the alert and counting the number of alerts that have been generated. So far, so good.
If you read our blog on Statistical Search, you may recall that we can group, or aggregate, by more than a single field or variable. When it comes to visualizing data over a time range, one of those values in the match section of your search will be a time field of some sort. The question then becomes, which time field? For detections, a good field to use is detection.detection_time.seconds.
$severity = detection.detection.severity
detection.detection.rule_set != ""
detection.detection.alert_state = "ALERTING"
$date = detection.detection_time.seconds
match:
$severity, $date
outcome:
$severity_count = count($severity)
order:
$date asc
However, if you add the detection time field and group the data on both the severity and time like we did above, the output may not be quite what you had in mind.
I mean, if you want the severity broken out by the second, this might be ok, but generally in a time chart, it’s more likely we are breaking this data out by hour or by day or some other time period. This brings us back to Google SecOps use of functions. There are a few different date and time functions that could be used like timestamp.get_hour, timestamp.get_date, heck, you could even use them together with strings.concat, but my go to function for this is timestamp.get_timestamp (video).
$severity = detection.detection.severity
detection.detection.rule_set != ""
detection.detection.alert_state = "ALERTING"
$date = timestamp.get_timestamp(detection.detection_time.seconds, "%F")
match:
$severity, $date
outcome:
$severity_count = count($severity)
order:
$date asc
By enclosing the detection timestamp with the function and the argument "%F", we are getting just the date from the detection time value. If you want it in another format or time measure, perhaps you are doing an hourly grouping in your chart, you could use "%k:00". The point is timestamp.get_timestamp and its arguments provide a good deal of customability.
Now our results show we had 56 low severity alerts on January 17, and six critical severity alerts the same day. Notice that for January 18th and 19th, we have severity counts broken out as well. Alright, at this point, we are ready to take our grouped search and convert this to a visual for our dashboard. We could leave it in this table format, just add a title and headings and click Add to dashboard and we’re done, but let’s see how we might visualize this.
Area, bar and line graphs all support grouping of result sets, so it’s as simple as choosing one of these options. Let’s start with the Area graph.
After naming the chart, we can scroll down the left side of the page and select the fields that will be represented on the X and Y-axis as well as which column will create the dimensionality by being grouped by. Google SecOps will populate these automatically but you can adjust them as you see fit. Generally, time will be across the X axis, a count or sum is on the Y axis and the group by value will be the field or variable that allows the data to be sliced in another manner. In this instance, the severity of the alert. We can see that on January 18, we had 29 high severity alerts generated. In the Data settings portion of the chart configuration, the different values in the group by field are represented by value and color. The colors can be modified if desired.
If we change our group type from Default to Stacked, we get a bit of a different view of the same dataset. In this view, we can easily see that we have a total of 100 alerts on January 18, with each band showing the number of alerts by severity appended to each other, rather than in the default view where each severity value started on the Y-axis at zero.
Finally, Percentile will display each value as a percentage of the daily volume. In this view, we can see that from January 17-19 that over 60% of the alerts were of low severity. This is a good example where we should update the label on our Y-axis to reflect that this is a percentage rather than a count so that someone viewing this chart in a dashboard doesn’t misinterpret what we are sharing.
The line chart works in the same manner as the area graph except rather than filling the area below the line with a color, the line itself is that color. The bar chart provides the same three grouping options with the Stacked type looking very similar to our example above.
The Default grouping will generate a color coded bar for each value within the day. In both of these views, we can see that there were seven critical severity alerts triggered on January 18.
As you can see, any of these could tell a story to the person reviewing the chart, which is our objective. Once you have the chart you want, click Add to dashboard.
My one piece of parting advice when building these kinds of charts that contain groupings like we’ve seen today is to think carefully about how you would like to slice the data to appear in the chart. In our example, we have four severities shown. However, with over 100 metadata.event_type values, that might get a bit messy. Limiting our visual to just event types that start with Network or Process or User might be a better option. Similarly, all the users won’t work well on the time chart but a filter, which we will get to, might be a good solution.