Log ingestion Best Pratices - Direct Ingestion logs

[ramonrisuenho]

Hi folks,

Recently, I got a SecOps implementation project that asked me to ingest some GCP-Native logs, such as CloudSQL, CloudNAT, Cloud Audit Logs and Cloud LOADBALACING (for Cloud Armor WAF) as well as logs for GKE and GCE. For GKE and GCE, I created log sink routing to a Cloud Storage and it is working fine after some replacement for the log queries, but, for the other ones I ingested using the Direct ingestion documentation but I didn`t understand how can I refine those logs in the Google SecOps configuration page.

They seem to be like an log query as the log sinks, but I couldn`t find any information or example to refine those logs. With the sugested standard log query we are getting way to much logs and we need to trim those down.

Here is how the query for those logs are configured:

"log_id("dns.googleapis.com/dns_queries") OR log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/policy") OR log_id("cloudaudit.googleapis.com/access_transparency")  OR log_id("dns.googleapis.com/dns_queries") OR log_id("compute.googleapis.com/firewall") OR log_id("ids.googleapis.com/threat") OR log_id("ids.googleapis.com/traffic") OR log_id("requests")  OR log_id("audit_log") OR log_id("recaptchaenterprise.googleapis.com/assessment") OR log_id("recaptchaenterprise.googleapis.com/annotation") OR log_id("run.googleapis.com/stderr") OR log_id("run.googleapis.com/stdout") OR log_id("run.googleapis.com/requests") OR log_id("run.googleapis.com/varlog/system") OR log_id("networksecurity.googleapis.com/firewall_threat")"

Could you guys help me with some clues or documentation? I would appreciate that!!

Thank you for your time!!

[Rene_Figueroa]

Hi @ramonrisuenho, you can add more filters to trim down the logs. For example, if you only wanted to exclude logs from a specific GCP project, you could have something similar to this:

(log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event")) AND
(NOT logName =~ "^projects/my-project-id/logs/.*$")

More samples can be found here:

https://cloud.google.com/chronicle/docs/ingestion/cloud/ingest-gcp-logs#export_filter_examples

If you need additional help with log query building, you can open a support case with the Logging team directly on GCP.