Syslog Parser Extension Syntax

[JuLenny]

Hi,

I'm using a Cisco Router and when I set initially the forwarder to include a CISCO_ROUTER forwarder, it provided very little information in the UDM.

I then I switched to a CISCO_VPN forwarder that didn't parse my log entry at all and then when I tried the CISCO_SWITCH forwarder, I'm getting more information, as you can see herein, but not the important information I need.

That's the RAW log:

<14>1 2024-11-10T21:23:25+08:00 router0ABCD log_sslvpnac - - [meta sequenceId="16"] facility=SslVpn;msg=INFO sslvpn_appl.c.146[747CF470] Client is Web Browser;

That's the UDM. As you can see, the msg data (that I need), wasn't parsed:

metadata.base_labels.allow_scoped_access = true
metadata.base_labels.log_types = "CISCO_SWITCH"
metadata.base_labels.namespaces = "LAN"
metadata.event_timestamp.seconds = 1731245064
metadata.event_timestamp.nanos = 229863000
metadata.event_type = "STATUS_UPDATE"
metadata.id = "AAAAAF2JL0NEDomaQVosIKN7W6cAAAAABgAAAA8AAAA="
metadata.ingested_timestamp.seconds = 1731245064
metadata.ingested_timestamp.nanos = 229863000
metadata.log_type = "CISCO_SWITCH"
metadata.product_event_type = "log_sslvpnac"
metadata.product_name = "Cisco Switch"
metadata.vendor_name = "Cisco"
principal.asset.hostname = "router0ABCD"
principal.hostname = "router0ABCD"
principal.namespace = "LAN"

Is it possible to add in the events section of the Rule Editor the ability to search for a string in the raw log itself, or am I forced to modify the parser (which I don't know how)?

Alternatively, is there another Cisco forwarder anyone can recommend that may yield a better outcome?

Thanks

[dnehoda]

You cannot use raw log in yara-l. 

You either need to modify the parser or submit a request to support with the specific fields you need to get that parsed correctly.  

[JuLenny]

Thanks @dnehoda 

So, This is the raw and UDM log that is partially parsed:

I want to create a parser extension:

What are the values we should put in the Syslog & Target fields to extract and unify the user, IP and login state?

Thanks

[cmorris]

Please see this - https://cloud.google.com/chronicle/docs/event-processing/using-parser-extensions#syslog_extractor

The syslog field is used to define the pattern to identify the Syslog header and the raw log message.

[JuLenny]

Thanks @cmorris . That's the doc I was looking for.

So, I built this filter, but it's being refused by the system:

%{TIMESTAMP_ISO8601} %{WORD:host} %{WORD:type} ([- ]+)?%{GREEDYDATA:meta} User %{WORD:user} %{WORD:status} from %{IP:ip}

The error I'm getting is:
generic::internal: failed to run grok filter: failed to parse data with all match patterns

What should I do? I want to extract into the metadata of the UDM the user, status and IP from this message and only for jsonproc type:
`

<14>1 2024-11-11T12:39:58+08:00 router0ABCD jsonrpc - - [meta sequenceId="6258"] User jlxx logged out from 10.0.0.142

Thanks

[dnehoda]

I posted the same doc below its  just down the page a little.  

Anyhow, something doesn't seem okay with your log.    I see 33 fields being parsed in my lab.   

metadata.base_labels.allow_scoped_access = true

metadata.description = "Logging to host 123.220.7.172 port 10519 started - reconnection"

metadata.enrichment_labels.allow_scoped_access = true

metadata.event_timestamp.seconds = 1731339409

metadata.event_timestamp.nanos = 571000000

metadata.event_type = "STATUS_UPDATE"

metadata.id = "AAAAANPjXxW0G/WoJhPsURDveV0AAAAABgAAAAAAAAA="

metadata.ingested_timestamp.seconds = 1731339415

metadata.ingested_timestamp.nanos = 257536000

metadata.prouct_event_type = "SYS-6-LOGGINGHOST_STARTSTOP"

metadata.product_name = "Router"

metadata.vendor_name = "Cisco"

principal.asset.ip = "123.220.7.172"

principal.ip_geo_artifact.ip = "123.220.7.172"

principal.ip_geo_artifact.location.country_or_region = "Japan"

principal.ip_geo_artifact.location.region_coordinates.latitude = 40.1376293

principal.ip_geo_artifact.location.region_coordinates.longitude = 140.334341

principal.ip_geo_artifact.location.region_latitude = 40.13763

principal.ip_geo_artifact.location.region_longitude = 140.33434

principal.ip_geo_artifact.location.state = "Akita"

principal.ip_geo_artifact.network.asn = "4713"

principal.ip_geo_artifact.network.carrier_name = "ntt communications corporation"

principal.ip_geo_artifact.network.dns_domain = "ocn.ne.jp"

principal.ip_geo_artifact.network.organization_name = "ntt communications corporation"

principal.ip = "123.220.7.172"

principal.location.country_or_region = "Japan"

principal.location.region_coordinates.latitude = 40.1376293

principal.location.region_coordinates.longitude = 140.334341

principal.location.region_latitude = 40.13763

principal.location.region_longitude = 140.33434

principal.location.state = "Akita"

principal.port = 10519

[dnehoda]

@JuLenny  - try that....sorry it was late last night and I couldn't respond.  

%{TIMESTAMP_ISO8601} %{WORD:host} %{WORD:type} ([- ]+)?%{GREEDYDATA:meta} User %{WORD:user} %{WORD:status} out from %{IP:ip}

[JuLenny]

Hi @dnehoda 
Thanks for your continued assistance on this.
I tried to input the string that you sent into the SYSLOG field, but may have done it incorrectly, as you can see herein:

Can you advise what's wrong here? Also, how do I make sure that all 3 fields that I want are included in the UDM metadata?
Thanks

[JuLenny]

Hi @dnehoda & @cmorris ,
I managed to build a Grok filter that extracts all the values that I need for that specific application (jsonrpc):

%{TIMESTAMP_ISO8601:timestamp} %{WORD:device} %{WORD:application} - - %{GREEDYDATA:meta} User %{WORD:user} %{GREEDYDATA:status} from %{IP:source_ip}

This is the output of the Grok Debugger:

Then, I tried to create a CBN snippet, using that Grok filter:

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:device} %{WORD:application} - - %{GREEDYDATA:meta} User %{WORD:user} %{GREEDYDATA:status} from %{IP:source_ip}" }
  }   
}

This is what it looks like:

But something in missing from my code, as during the validation, it tries to run on the last message received from the switch (not jsonrpc) and since the syntax is different, it fails:

I tried to review the online guide for this, but didn't understand what to put in the code to dedicate this Grok filter to the specific jsonrpc application.

What am I doing wrong? I feel that I'm close, but I don't know how to proceed.
Thanks for your help

[JuLenny]

Hi again everyone
Does anyone here has any experience with parser extensions for syslog messages? I couldn't find anything relevant in the documentation.
Thanks

[JuLenny · Accepted solution]

I managed to create this extension that I could validate. It doesn't breakdown the fields like I wanted, but using RegEx afterwards does yield the outcome I need:

filter {

  mutate {

    replace => {

      "syslog" => ""

      "description" => ""

    }

  }

 

  grok {

    match => {

        "message" => ["(?<syslog>.*?)\[meta sequenceId=\"\d+\"\]\s*(?<description>.*)"]

    }

    overwrite => ["syslog", "description"]

  }

if [description] != "" {

    mutate {

        replace => {

            "event.idm.read_only_udm.metadata.description" => "%{description}"

        }

    }

}
 

#statedump{}

  mutate {

    merge => {

        "@output" => "event"

    }

  }

}

I hope that it will be useful to other members of this community 

[dlove40]

You can also create a parser extension to parse our the value of msg. That way you are not customizing a standard parser.

[JuLenny]

Yes. That's exactly what I'm trying to do

[dlove40]

Sorry. The message approval process didn't show your message about trying to create a parser extension when I made my comment. Only your initial post and the one from dnehoda.

[dnehoda]

Okay so, you're sharing two different log sources in your examples.   One is sequence ID 14 and the other is sequence ID 2433.  Lets try to clear that up.

There's certain considerations when you build an extension.  Also, you need to understand how that data is being sent over to SecOps.  In this specific case, and potentially in all Cisco Router/switch logging, its sending via syslog or syslog/kv.   I don't know just guessing here - but it seems maybe you are sending this to an intermediary syslog server and its chopping up the log somehow.  maybe not.  

Regardless though and this is a process that isn't typically handled well through messaging but:

Identify each field that you require to be parsed out - what else are you trying to get out of that log?  There's not much there.  Assuming you want the IP Address.  

Second you need to create a grok statement to get that properly formatted and brought out within the parser to UDM. 

There's examples here:

https://cloud.google.com/chronicle/docs/event-processing/using-parser-extensions

[JuLenny]

Hi @dnehoda 

You're right. I found another log (jsonproc) that provided more info that I preferred, and I'm focused on it. To answer you question, please refer to my response to @cmorris . I want to extract the user name, connection status and source IP of his connection.

Cheers

[apande1]

Hi @JuLenny 

I stumbled upon your post to find the grok syntax you were looking for , I had something similar built up for cisco switch.. made changes as per your need posting it here..
If you have logs with different stanza's you can always add  multiple grok patterns.

filter {

     mutate{

        replace => {

            #"_raw" => ""

            "Hostname" => ""

            "Date" => ""

            "Summary" => ""

            "Message"=> ""

            "connection"=> ""

            "Msg" => ""

            "SeqID" => ""

            "IP" => ""

            "USERID" => ""

        }

     }

    mutate {

        gsub => ["message", "\n", "LINE_BREAK"]

    }

    mutate {

        gsub => ["message", "LINE_BREAK", "\n"]

    }

    grok{

        match => {

            "message" = [

               "<%{INT}>%{NUMBER} %{TIMESTAMP_ISO8601:Date} %{GREEDYDATA:Hostname} %{WORD:application} - - \[meta sequenceId=\"%{GREEDYDATA:ID}\"] User %{USERNAME:USERID} %{GREEDYDATA:Message} %{IPV4:IP}"

               "<%{INT}>%{NUMBER} %{TIMESTAMP_ISO8601:Date} %{GREEDYDATA:Hostname} %{WORD:application} - - \[meta sequenceId=\"%{GREEDYDATA:SeqID}\"] facility=%{GREEDYDATA:connection};msg=%{GREEDYDATA:Msg}"

            ]

        }

        overwrite = ["Hostname", "Date", "USERID", "IP", "Summary", "Message", "connection", "Msg", "SeqID"]

        on_error => "invalid_grok"

    }

    mutate{

        replace => {

            "Src_IP_temp" => "%{IP}"

        }

        convert => {

            "Src_IP_temp" => "ipaddress"

        }

        on_error => "not_a_src_ip"

    }

    if ![not_a_src_ip]{

        mutate{

            merge =>{

                "event.idm.read_only_udm.principal.ip" => "IP"

            }

        }

    }

    if [Msg] != ""{

        mutate{

            replace => {

                "event.idm.read_only_udm.metadata.description" => "%{Msg}"

            }

            on_error => "no_msg"

        }

    }

    mutate {

        replace => {

            "event.idm.read_only_udm.metadata.product_name" => "Cisco Switch"

            "event.idm.read_only_udm.metadata.vendor_name" => "Cisco Switch"

            "event.idm.read_only_udm.target.hostname" => "%{HOSTNAME}"

        }

        on_error => "multiple_replace_error"

    }

    mutate {

        merge => {

            "@output" => "event"

        }

    }

}

you can add other fields and give them UDM values as per your choice.. Its little late but, hope this helps.