How to avoid duplication of cases/events/alerts ?

[shubhamagar]

I have created a custom connector in Google Secops SOAR but whenever the connector runs it re-creates even those cases which it has already created in previous run. How this could be avoided ? 

Is there any way by which unique value of each event/alert can be matched to existing cases/alerts/events in SOAR ? What will be the optimized way to achieve this ?

[f3rz]

Hi @shubhamagar, first, you can try to find this unique value from the source system from where you ingest alerts (e.g., alertId, messageId (for email), etc.).

Then, you can use read_ids and write_ids, to save those IDs to SOAR DB and retrieve them from there on the next run to compare against newly ingested alerts. You can find how it is implemented for example in Splunk Query Connector in IDE.

Also, here you can find how to find latest version of TIPCommon and add it to your custom integration: https://www.googlecloudcommunity.com/gc/SOAR-Forum/TIPCommon-Library-Installation/m-p/735638/highlig...

[shubhamagar]

Hi,

Thank you for your response. However, it seems that read_ids and write_ids do not persist. In the next connector run, when read_ids is used, it returns an empty list with the message: Returning default value instead: [ ]

[f3rz]

Hi @shubhamagar, if everything is configured correctly, IDs should be persistent. 

In the case of a local connector, they should be saved to DB, and in the case of a Remote Agent connector, they should be saved to the file on the Remote Agent file system. 

If this doesn't work for you, I recommend you to see if your code is written in the same way as commercial connectors and if the issue is still there - contact Support. 

[SoarAndy]

As per f3rz, but just to expand 1%

Some vendor API calls allow you to query 'any alert after 123', in this case you track that the last thing you ingested was 123

Some vendors track the last ingest timestamp, so you would need to track the last window you ingested

Some vendors don't do any of this, and just return a list of the last 100, in this case you need to track what you ingested

And then some vendors give you a list but don't tell you which Alerts had new things added, that's even more fun 😄 !

It really is vendor specific unfortunately