Json parsing inside Message

[Former Community Member]

Hello Team, 
we are trying to parse fields from json log format, but there are nested fields  Within the "Message".

we user the json filter and other different steps.

All the fields inside the "Message" are not being parsed, Please chek the sample log for reference.
{"EventReceivedTime": "202*-0*-** 21:44:45","SourceModuleName": "in_json_log","SourceModuleType": "im_file","Message": ",{\"_id\":\"66**c7d23afb**6e6*0f2\",\"serialNum\":0,\"acknowledged\":***,\"time\":\"202*-0*-**T14:14:04.657Z\",\"hostname\":\"e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379",\"fqdn\":\"\",\"containerName\":\"bu\",\"containerID\":\"2a91f881c8075c4f34db5*****3d6e\",\"imageName\":\"d***.io/google/cloud-:l\",
 please suggest a way to parse the fields insite Message.

[bsalvatore · Accepted solution]

Hi,

This is the real message received in the SIEM (escaped character, comma character before { , etc.)? Because if the original message like the following:

{
    "EventReceivedTime": "202*-0*-** 21:44:45",
    "SourceModuleName": "in_json_log",
    "SourceModuleType": "im_file",
    "Message": {
        "_id": "66**c7d23afb**6e6*0f2",
        "serialNum": 0,
        "acknowledged":"***",
        "time": "202*-0*-**T14:14:04.657Z",
        "hostname": "e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379",
        "fqdn": "",
        "containerName": "bu",
        "containerID": "2a91f881c8075c4f34db5*****3d6e",
        "imageName": "d***.io/google/cloud-:l"
    }
}

 you can easy extract it using the json command:

  json {
    source => "message"
    array_function => "split_columns"
    on_error => "not_json"
  }

 and the parser automatically create the id, serialNum, hostname, etc. variable.

[bsalvatore · Accepted solution]

Hi,

This is the real message received in the SIEM (escaped character, comma character before { , etc.)? Because if the original message like the following:

{
    "EventReceivedTime": "202*-0*-** 21:44:45",
    "SourceModuleName": "in_json_log",
    "SourceModuleType": "im_file",
    "Message": {
        "_id": "66**c7d23afb**6e6*0f2",
        "serialNum": 0,
        "acknowledged":"***",
        "time": "202*-0*-**T14:14:04.657Z",
        "hostname": "e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379",
        "fqdn": "",
        "containerName": "bu",
        "containerID": "2a91f881c8075c4f34db5*****3d6e",
        "imageName": "d***.io/google/cloud-:l"
    }
}

 you can easy extract it using the json command:

  json {
    source => "message"
    array_function => "split_columns"
    on_error => "not_json"
  }

 and the parser automatically create the id, serialNum, hostname, etc. variable.

[Former Community Member]

Hi bsalvatore,

I tried the json filter, it dosent work for the fileds inside Message.

{
    "EventReceivedTime": "202*-0*-** 21:44:45",
    "SourceModuleName": "in_json_log",
    "SourceModuleType": "im_file",

These fields will work with the json filter 

but the fields inide Message will not get parsed.

"Message": {
        "_id": "66**c7d23afb**6e6*0f2",
        "serialNum": 0,
        "acknowledged":"***",
        "time": "202*-0*-**T14:14:04.657Z",
        "hostname": "e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379",
        "fqdn": "",
        "containerName": "bu",
        "containerID": "2a91f881c8075c4f34db5*****3d6e",
        "imageName": "d***.io/google/cloud-:l"
    }

[bsalvatore · Accepted solution]

have you tried to use the statedump {} command to debug the message? If the raw log is correct, you should view like the screenshot in the debug view

In this case you can simply access to data using the following block (I suggest you to check if a text are mapped):

  if [Message][containerID] != "" {
          mutate {
        replace => {
          "event.idm.read_only_udm...." => "%{Message.containerID}"
        }
      }
  }

[Former Community Member]

I have user the statedump, the fields are getting mapped.

I will use the above syantx, hopefully it works