Training for Custom Parser

[TPankaj]

Hi Team,

Could you please provide guidance on how we can begin training to create a custom parser?

I have gone through below documentation, any other documenation will be helpful.

https://cloud.google.com/chronicle/docs/reference/parser-syntax

[kentphelps]

You may want to take a look at the sections on Ingestion and normalization in the Cloud Skills Boost SecOps training

[TPankaj]

Thank you @kentphelps !

[Ben_T]

In addition to the parsing content in the Skillboost training, there is some great content in our community adoption guides. I've linked the one for UDM Parsing Deep Dive below:

https://www.googlecloudcommunity.com/gc/Adoption-Guides/Security-Operations-Deep-Dive-into-UDM-Parsi...

[TPankaj]

Thank you @Ben_T !

[Old_Wolf]

I don't know if this will help you.

(11) Chronicle Coalface: Writing a Parser Extension | LinkedIn

[AJMSecOps]

What format are the logs in? My approach is generally to get a sample of the logs you'll be ingesting, understand the format, fields available, and how those fields map to UDM.

[Old_Wolf]

I appreciate your approach. If I can, I like to get the logs into something like CRIBL or Bindplane first so that I can confirm the data coming in.  Next, I will forward them to Chronicle, with the appropriate log type for parsing, and a namespace, and then check how well the logs are parsed and what additional information I want to extract from the logs.

Syslog: RFC 5424 does not have the severity level, e.g. notice, in the event like RFC 3164 did, so you have to derive it from the PRI number. I could not do the math I wanted in a parser extension, which is why I created the extension I shared in the link I shared. I add this to all my syslog feeds as an extension so I can access the data.

[AJMSecOps]

Oh ok sounds like you're not writing a parser from scratch then. For something like this, I get the log into SecOps, do a raw log search, look at a sample of an unparsed log, click on Manage Parser --> Create Custom Parser. From there, I would just evaluate what field you want to extract (via Grok) and assign to your desired UDM field. Can use statedump to ensure the field is being stored properly as SecOps expects it. 

Sorry without looking at the specific log and exact UDM mapping you want to do, it's a bit difficult to be more helpful.

[Old_Wolf]

Sorry, I mis-read at the beginning. Try looking at this.

Chronicle Coalface: Writing a Custom Parser | LinkedIn

Statedump is useful for seeing how things are being extracted and how things are being placed into udm fields.

Don't forget to comment them out, #, before you validate or the validation will fail.