Repeated fields overwritten in parser extension (code)

[chrisd2]

Hello everyone,

I wrote a parser extension ("code" mode) for a log_type in order to add a couple fields that were not handled by the default parser.
I mapped a couple raw fields to UDM fields under security_result in the parser ext.

Problem: I noticed that this configuration was overwriting the security_result field parsed by the default parser.

I saw in the docs that in "no-code" parser extensions, there was the option to append values to repeated fields instead of overwriting them. How can I configure the same behavior for my "code" parser extension ?

[cbryant]

You can create a field mapping extension within the parsers UI in the SecOps tenant, the same way you likely did with the CBN code method. Switch the method radio button to "Map data fields" and the repeated field radio button to "Append Values". From there you will see the following:

1. These tabs relate to the specific mapping conditions you create. You can switch between them, add, or remove as needed.
2. Precondition field, the field name to evaluate a condition on. 
3. Operator on the precondition field (equals or not equals).
4. The value you are looking to compare the precondition field's value to. (leave 2-4 blank to omit the condition requirement)
5. The name of the field to map to UDM on a true condition evaluation.
6. The destination UDM field. 

Summary of what this example is doing: if the precondition field "foo" contains the value "bar", map the value in "bin" to metadata.description. This results in the value "baz" being mapped to metadata.description when this extension is run. 

[chrisd2]

Hello, the docs state that we cannot use "Map Data fields" extensions for repeated fields that are not at the lowest level of the hierarchy :

Important: Data field mapping instructions can include repeated fields only when the repeated field is at the lowest level of the hierarchy. For example, you can map values to the udm.principal.ip repeated field because ip is at the lowest level of the hierarchy and principal is not a repeated field. You cannot map values to udm.intermediary.hostname because intermediary is a repeated field and is not at the lowest level of the hierarchy.

No chance with this parser extension mode neither for security_result 😞

[chrisd2]

Hello @cbryant ,

Thank you for your help.

I saw the radio button that allows to "Append values" to repeated fields for the "Map data fields" method.

Unfortunately I would really prefer to use the "Write code snippet" method. How can I make a CBN parser extension append the values to repeated fields instead of overwriting them ?

[cbryant · Accepted solution]

To my knowledge, there is no way to append repeated fields with a CBN snippet. 

[cmorris]

@chrisd2 - Can you share the snippet you are working with now that is overwriting the security_result field?

[chrisd2]

Hello @cmorris 

Here is a brief version of the parser extension :

filter {

  mutate {
    replace => {
      "event" => ""
      "sec_result" => ""  # Temporary token used to map log fields to security_result UDM field
    }
  }

  # {...} Data extraction logic redacted, all raw log fields go in the 'event_log' token and its sub-tokens

  if [event_log][pagerisk] not in ["", "None"] {
    mutate {
      replace => {
        "sec_result.risk_score" => "%{event_log.pagerisk}"
      }
      convert => {
        "sec_result.risk_score" => "float"
      }
    }
  }

  if [event_log][app_risk_score] not in ["", "None"] {
    mutate {
      replace => {
        "app_risk_score_label" => ""
      }
    }
    mutate {
      replace => {
        "app_risk_score_label.key" => "app_risk_score"
        "app_risk_score_label.value" => "%{event_log.app_risk_score}"
      }
    }
    mutate {
      merge => {
        "sec_result.detection_fields" => "app_risk_score_label"
      }
    }
  }

  if [event_log][ruletype] not in ["", "None"] {
    mutate {
      replace => {
        "ruletype_label" => ""
      }
    }
    mutate {
      replace => {
        "ruletype_label.key" => "ruletype"
        "ruletype_label.value" => "%{event_log.ruletype}"
      }
    }
    mutate {
      merge => {
        "sec_result.detection_fields" => "ruletype_label"
      }
    }
  }

  if [sec_result] != "" {
    mutate {
      merge => {
        "event.idm.read_only_udm.security_result" => "sec_result"
      }
    }
  }

  # Output data to UDM
  # ------------------
  if [event] != "" {
    mutate {
      merge => {
        "@output" => "event"
      }
    }
  }
}

[chrisd2]

Hello,
Just to add to the conversation, for future readers, the "Map data fields" extension with "Append Values" to Reapeated Fields is not a solution to avoid overwriting of security_result by the parser extension.

Indeed, from the docs :

Important: Data field mapping instructions can include repeated fields only when the repeated field is at the lowest level of the hierarchy. For example, you can map values to the udm.principal.ip repeated field because ip is at the lowest level of the hierarchy and principal is not a repeated field. You cannot map values to udm.intermediary.hostname because intermediary is a repeated field and is not at the lowest level of the hierarchy.

>>> security_result is also not at the lowest level of the hierarchy.