Error in parsing

[asinghz297]

So I havce written a parser.
It's a custom parser.

filter {

  mutate {

        replace => {

            "raw_Event" => ""      

        }

    }

    grok {

        match => {

            "message" => ["%{GREEDYDATA:raw_Event}"]

        }

        overwrite => ["raw_Event"]

        on_error => "grok_failed"

    }

      if "raw_Event" != "" {

    mutate {

        replace => {

          "additional_fields.key" => "raw_Event"

          "additional_fields.value.string_value" => "%{message}"

        }

    merge => {

        "event1.idm.read_only_udm.additional.fields" => "additional_fields"

      }

    }

  }

    mutate {

        merge => {

            "@output" => "event1"

        }

    }

    statedump{

        label => "first"

    }

}



The error I am facing states : 

generic::unknown: enrichment failed for event 0: LOG_PARSING_GENERATED_INVALID_EVENT: "generic::invalid_argument: UDM.metadata not present"



Can someone help me resolve this?

[citreno]

You will need to set at minimum metadata.event_type with one of the allowed values

You probably want to use GENERIC_EVENT as the other types have more validation. That said I wouldn't design a parser this way, so please note a couple of things:

1. Udm is not intended to cram raw logs into arbitrary fields and there's some back end validation logic that may block you and there's very little benefit of doing it since there's raw log search capabilities anyways that will be faster and more intuitive.

2. It is considered a good parser design to actually extract the fields within the logs and fit them into appropriate fields in the udm schema and use appropriate  event_types. if that's not done the capabilities of the platform are limited to simple raw log searches with regex and what not.

Hope this helps!

[asinghz297]

@citreno 

I want a field to have the whole raw event.

filter {

  mutate {

        replace => {

            "raw_Event" => ""  
            "vendor_name" => "Indusface"
            "product_name" => "WAF" 

        }

    }

    grok {

        match => {

            "message" => ["%{GREEDYDATA:raw_Event}"]

        }

        overwrite => ["raw_Event"]

        on_error => "grok_failed"

    }

      if "raw_Event" != "" {

    mutate {

        replace => {

          "additional_fields.key" => "raw_Event"

          "additional_fields.value.string_value" => "%{message}"

        }

    merge => {

        "event.idm.read_only_udm.additional.fields" => "additional_fields"

      }

    }

  }

    mutate {

        merge => {

            "@output" => "event"

        }

    }

}




Can you guide me through it? How to set at minimum metadata.event_type?

[bsalvatore]

Hi,

I suggest you to follow this page Important UDM fields for parser data mapping  |  Google Security Operations  |  Google Cloud to map the important UDM fields, because there are some of it used in the SIEM to enrichments the data and inside curated detection, for example, the

<event>.principal.ip

are used in the following areas: Curated detections, Indexing, Artifact aliasing, Asset aliasing.

[asinghz297]

generic::unknown: enrichment failed for event 0: LOG_PARSING_GENERATED_INVALID_EVENT: "generic::invalid_argument: UDM.metadata not present"


Can you tell me what do I need to change exactly to fix this. 

@jstoner @bsalvatore @citreno 

[bsalvatore]

very very simple solution:

filter {
  mutate {
        replace => {
            "raw_Event" => ""  
            "event.idm.read_only_udm.metadata.vendor_name" => "Indusface"
            "event.idm.read_only_udm.metadata.product_name" => "WAF" 
            "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
        }
    }

    grok {
        match => {
            "message" => ["%{GREEDYDATA:raw_Event}"]
        }
        overwrite => ["raw_Event"]
        on_error => "grok_failed"
    }

      if "raw_Event" != "" {
    mutate {
        replace => {
          "additional_fields.key" => "raw_Event"
          "additional_fields.value.string_value" => "%{message}"
        }
    merge => {
        "event.idm.read_only_udm.additional.fields" => "additional_fields"
      }
    }
  }

    mutate {
        merge => {
            "@output" => "event"
        }
    }

}