LinkedIn
Twitter
In Google Security Operations (SecOps), single-column reference lists have been a longstanding method for including or excluding events in detection rules based on a list of strings, regex expressions, or CIDR ranges. But what if you need to apply more complex logic in your rules that involves filtering events based on multiple criteria or enriching events/entities using custom data? We recently launched data tables in public preview to help security teams with their more sophisticated event filtering and enrichment use cases.
A data table like the example shown below is made up of named columns (in the first row of the table) and rows of data. Each column of a data table must be mapped to either a data type (string, regex, or CIDR) or to a UDM entity field (e.g. entity.user.termination_date.seconds). Specifying a data type for a column in a data table allows you to use values in those columns to filter events in your rules. Mapping a data table column to an entity field allows you to override, append, and exclude data in Google SecOps’ Entity Graph using custom data.
This post will demonstrate how to create data tables and use them to filter events in YARA-L rules. In future posts, we’ll do a deep dive on how to write values to a data table using the results from a rule and how to use a data table to modify data in the entity graph, but we’ll “table” that for now 😐
Please note, while the data tables feature is in public preview, it’s not possible to read & write data to/from data tables in search queries. This functionality is expected to be added in a future release.
Now that we know what data tables are and how they can be used at a high level, I’ll go ahead and create a new data table and show how to use it in a rule.
From the Data Tables page in Google SecOps, I click the “Create” button and enter a name and description for the new table. A common detection use case for security operations teams is to maintain a list of users to monitor for suspicious activity such as data exfiltration. This is the example use case that we’re going to focus on today.
The new data table has been created. Now it’s time to populate it with some data. I can either manually type the data (column headers and rows) into the user interface or I can click the “Import File” button to upload a CSV/TSV file to populate the data. I’m going to upload the following CSV file.
user_email,expiry_date
mikeross@cymbal-investments.net,2025-05-01 00:00:00
roxysmith@cymbal-investments.net,2025-05-01 00:00:00
rachelmason@cymbal-investments.net,2025-06-01 00:00:00This is a small data table for demonstration purposes. We will explore more complex use cases for data tables as this blog series progresses.
After reviewing the import file options, I click “Import Data” and my new data table is populated with the data from my CSV file.
Our next step is to define the data type (STRING, REGEX, or CIDR) for each of the columns in the data table. Setting the data type to REGEX or CIDR allows for the comparison of values in events that you’ve ingested into SecOps and regular expressions or CIDR IP address ranges stored in the data table. Setting the data type to STRING allows you to do row- or column-based comparisons based on string values.
Selecting the STRING data type is fine for both of these columns. The “expiry_date” column contains a date and time that we will convert to a timestamp in the logic for the new rule that we’re about to create.
The rule shown below is a customized version of one of our community rules. It detects when a user who is in the “monitored_users” data table shares a file via Google Drive with an email address that’s associated with a free service such as Gmail or Hotmail. I’ve removed some values from the meta and outcome section of the rule for brevity.
Specifically, the rule does the following:
- Filters Google Workspace events to identify Google Drive file share events
- Performs a row-based comparison between the user’s email address and the values in the “user_email” column of the “monitored_users” data table
- Checks that the event timestamp (metadata.event_timestamp.seconds) is earlier than the “expiry_date” for the user in the data table
Let’s take a closer look at the YARA-L syntax for performing a row-based comparison between a value in a UDM event and the values stored in a data table.
rule monitored_user_google_workspace_file_shared_from_google_drive_to_free_email_domain {
meta:
author = "Google Cloud Security"
description = "Identifies when a user account that is on our list of monitored users shares a file that's stored on Google Drive with a free email domain."
events:
$workspace.metadata.vendor_name = "Google Workspace"
$workspace.metadata.product_name = "drive"
(
$workspace.metadata.product_event_type = "change_user_access" or
$workspace.metadata.product_event_type = "change_document_visibility" or
$workspace.metadata.product_event_type = "change_document_access_scope" or
$workspace.metadata.product_event_type = "change_acl_editors"
)
// File shared with an email address that's associated with free email service
// This list of domains can be customized and/or stored & maintained in its own data table
$workspace.target.resource.attribute.labels["visibility"] = "shared_externally"
$workspace.target.user.email_addresses = /.*@gmail\.com|.*@aol\.com|.*@ymail\.com|.*@ymail\.com|.*@hotmail\.com|.*@outlook\.com|.*@icloud\.com/
$user_email = $workspace.principal.user.email_addresses[0]
// Check if the user's email address is found in a row within the data table's 'user_email' column
$user_email = %monitored_users.user_email
// Ensure that the event timestamp is earlier than the 'expiry_date' that's stored for the user in the data table
$workspace.metadata.event_timestamp.seconds < timestamp.as_unix_seconds(%monitored_users.expiry_date)
match:
$user_email over 15m
outcome:
$target_emails = array_distinct($workspace.target.user.email_addresses)
$doc_name = array_distinct($workspace.target.resource.name)
condition:
$workspace
}In the rule’s events section, we are looking for events where the user’s email address (stored in the $user_email placeholder variable) is found within a row in the data table for the “user_email” column. The syntax for referencing a data table column is %data_table_name.column_name.
We’re also comparing the timestamp (metadata.event_timestamp.seconds) for the UDM event with the timestamp that’s stored in the “expiry_date” column for the same user. Note that the timestamp.as_unix_seconds function is being used to convert the timestamp (e.g. 2025-04-01 00:00:00) to an epoch timestamp value ready for comparison with the timestamp in the UDM event.
I’ve used the term, row-based comparison a couple of times. When working with data tables, a row-based comparison is used to evaluate conditions for a single row in a data table. In the example rule above, we want to ensure that the values for a user’s email address and “expiry_date” are evaluated against a single row in the data table. We don’t want to match a user’s email address in one row in the data table and match the event timestamp with another user’s “expiry_date” in a different row within the data table.
Row-based comparisons are performed by using equality operators such as = and <. For example, $user_email = %monitored_users.user_email. Column-based comparisons are performed by using the “in” keyword. You can read more about row- and column-based comparisons in the documentation.
Testing the rule by running it over the last two weeks of events reveals that user “mikeross@cymbal-investments.net” shared a file, “Customer Proposals - March 2025” via Google Drive with a Gmail email address.
Wrap up
In this post, we learned how to create data tables in Google SecOps and populate them with data. We walked through an example of how to utilize data tables in YARA-L rules to filter events based on a detection use case. I also explained the difference between row- and column-based comparisons when using data tables.
I hope you found this useful. Stay tuned for more posts on data tables in the near future. I’ll be providing examples on how to write results from a rule to a data table, how to enrich entities using custom data, and how to manage data tables using Google SecOps’ API.