Best way to export events from broad queries

[wonjulee]

Hi,

What is the best way to export large amounts of results from (very) broad queries? It seems that the SecOps UI only shows and allows the exporting of 1,000,000 events when very broad queries are executed. I want to export a large amount of parsed event data (more than 20 million individual events) and analyse the data through pandas on python. 

I understand that I can do this through the SecOps API or through BigQuery, but are there any costs incurred through these processes? Would there be any more efficient methods that I could try out?

Thank you very much.

[gkush · Accepted solution]

Hi Wonjulee,

As part of the shared resource of the SecOps SaaS, SecOps limits your queries, whether UI based or API-based, to 1M results in a UDM Search. This doesn't mean that aggregate information gets cut off at 1M records. Suppose you wanted a count of all DNS lookups done across all endpoints for a month?  In most places that would reach past 1M results, and with Stats Search you could get the aggregate count, but not the full set of data.

Since you said you wanted the records themselves for analysis, you could try to break your query up into chunks and align based on timestamps.  That would be something you could try in a script - get a set, figure out the timestamp, keep looking backwards from the last returned result.

Depending on your license tier, another option might be the BigQuery data lake?  You should be able to run your SQL-based query on the UDM fields you want and then pull the data out.  

[AymanC]

Hi @wonjulee,

It may be worth looking into the following API endpoint: https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/projects.locations.instances.legacy/l...

I've not personally tried this, but our Customer Success Team has previously suggested using it to export a large amount of data.

Kind Regards,

Ayman