Assistance Needed for Parsing JSON Array in Google Security Operations

[harshwardhan16]

Hi all,

I am new to Google Security Operations and am currently working on creating a custom parser. However, I am encountering issues when trying to create a UDM for a JSON array that lacks a parent object, which is preventing me from looping through it.

Input Json

[

{

"serviceId": 1012,

"assetType": "Consumer",

"shortVin": 64,

"vehicleModel": 436,

"affectedAssets": "KH35"

},

{

"serviceId": 4564,

"assetType": "Consumer",

"shortVin": 234,

"vehicleModel": 46,

"affectedAssets": "N435"

}

]

Error
generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter json (0) failed: invalid JSON: json: cannot unmarshal array into Go value of type map[string]interface {}"generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter json (0) failed: invalid JSON: json: cannot unmarshal array into Go value of type map[string]interface {}"

Any guidance or help on how to resolve this issue would be greatly appreciated.

Thank you!

[Shubhamgarang]

Hi @harshwardhan16  

You can loop for this hope this will resolve your issue .

[harshwardhan16]

Hi @matthewnichols / @deeshu 

Can you help me in this problem i try some method but nothing work. 

Any guidance or help on how to resolve this issue would be greatly appreciated.

Thanks 

[matthewnichols]

Hi @harshwardhan16 Apologies, I've been out of office the last week. Let me find someone that might be able to help answer your question. 

[harshwardhan16]

Thanks

[dnehoda]

What does your custom parser look like?

also, you can use a <statedump> on the 2nd to last line of your parser config to see where the problem lies.  

[harshwardhan16]

hi @dnehoda ,
I try code similer to this 

filter {

json {
source => "message_1"
array_function => "split_columns"
on_error => "not_in_json_format"
}
mutate {
replace => {
"event1.idm.read_only_udm.metadata.product_name" => "Upstream"
"event1.idm.read_only_udm.metadata.vendor_name" => "Upstream"
"event1.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
"event1.idm.read_only_udm.metadata.product_version" => "%{json_data.source}"
}
}
#TODO(evalute a more specific UDM event type)

mutate {
merge => {
"@output" => "event1"
}
}
}

error is that given data is not a json format. 

[AbdElHafez]

Hi @harshwardhan16 ,

The input message is a list not a JSON, that is why the parser is throwing away an error because the json clause expects a json object, the split clause is for json values that are lists within the root json object, not for a list input.

Before I modify your parser, do you need your parser to be able to handle -or expect it to receive- list input like that ? and do you need a single event or multi-event output ?

[AbdElHafez]

I modified my last post, this should be fine, you could assign the tokens of "v" to other fields instead of having all of them in "metadata.product_version" ;

filter {

mutate {

        gsub => [

            "message","\\[","",

               ]

        on_error => "subError"

    }

mutate {

        gsub => [

            "message","\\]","",

               ]

        on_error => "subError"

    }

mutate {

        gsub => [

            "message","\\n","",

               ]

        on_error => "subError"

    }

mutate {

        gsub => [

            "message","\\},\\{","}|{",

               ]

        on_error => "subError"

    }

    mutate {

        split => {

        source => "message"

        separator => "|"

        target => "messageSplit"

        }

    }

    for k,v in messageSplit {

   mutate {

     convert => {

       "k" => "string"

     }

   }

    mutate {

    replace => {

    "event" => ""

    }

    }

    json {

    source => "v"

    array_function => "split_columns"

    }

    mutate {

    replace => {

    "event.idm.read_only_udm.metadata.product_name" => "Upstream"

    "event.idm.read_only_udm.metadata.vendor_name" => "Upstream"

    "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"

    "event.idm.read_only_udm.metadata.product_version" => "%{v}"

    }

    }

    mutate {

    merge => {

        "@output" => "event"

    }

    }

statedump {

    "label" => "afterLoop"}

    }

}