Cloud Function MISP ingestion

[samnewton]

Hi Everyone,

I've set up a Cloud Function to ingest MISP feeds into my Chronicle SIEM instance. I'm getting some failures uploading the events which I'm assuming is due to the 1MB limit (the smaller feeds upload without a problem). In this post: https://medium.com/@thatsiemguy/how-to-integrate-misp-and-chronicle-siem-9e5fe5fde97c he mentions a Python code snippet to break it into manageable pieces.

I'm assuming this needs to get added to my main.py file, but I'm having no luck getting it to work. 

Has anyone had any luck with this?

< # Iterate through all the events and ingest data into Chronicle.
< for data in response_events.get("response", []😞
< event_json = data.get("Event", {})
<
< # Remove unwanted key-value and append the
< # updated dictionary to data_list.
< updated_dict = {
< key: event_json.get(key)
< for key in event_json
< if key not in KEYS_TO_REMOVE
< }
< data_list.append(updated_dict)
---
> dict_example = {}
>
> for response in response_events["response"]:
> for key, value in response.items():
> for key2, value2 in value.items():
> if type(value2) == type(str()):
> dict_example[key2] = value2
> if key2 == "Org":
> dict_example[key2] = value2
> if key2 == "Orgc":
> dict_example[key2] = value2
> if key2 == "Attribute":
> for idx, attr in enumerate(value2):
> new_dict = {}
> new_dict = dict_example.copy()
> new_dict["Attribute"] = attr
> new_dict_json = json.dumps(new_dict)
> data_list.append(new_dict)
> if key2 == "Object":
> for i in range(len(value2)):
> for key3, value3 in value2[i].items():
> if type(value3) == type(str()):
> dict_example[key3] = value3
> if key3 == "Org":
> dict_example[key3] = value3
> if key3 == "Orgc":
> dict_example[key3] = value3
> if key3 == "Attribute":
> for idx, attr in enumerate(value3):
> new_dict = {}
> new_dict = dict_example.copy()
> new_dict["Attribute"] = attr
> data_list.append(new_dict)

Thanks in advance

Sam

[ion_]

What issues are you getting? Are you seeing 400s when you send it to the chronicle API?

[samnewton]

Hi @ion_ 

Yes, I'm seeing the following:

RuntimeError: Error occurred while pushing logs to Chronicle. Status code 400. Reason: {'error': {'code': 400, 'message': 'Request contains an invalid argument.', 'status': 'INVALID_ARGUMENT'}}

[cmmartin_google · Accepted solution]

A copy of the Cloud Function is available here:

https://github.com/goog-cmmartin/thatsiemguy/tree/main/misp

Alternatively, you can use the Extended CSV format with a the default MISP parser.  I can share an example of how to implement if that's be prefereable.

The 400 will be an error from the MISP rest API, so you may find more on the MISP server admin / api logs from memory.

[samnewton]

Thanks @cmmartin_google by using the config files provided in the above link I've managed to get it working. I appreciate the help.