Raw logs (ingestion API) + custom Parser

[SoarMike]

Good morning,

I have a question about log ingestion via the ingestion API. Initially, my logs contained only 13 fields for testing purposes. However, I later expanded them to include 22 fields. I built a custom parser, and when I preview it, the UDM output appears correct.

The problem I am having now is when I go to validate the parser it errors out and indicates that the older raw log with 13 fields is the cause which makes sense as the columns/fields are missing.

I was wondering is there a way to remove an individual log or even just remove all entries and start fresh?

Thanks in advance

[bsalvatore · Accepted solution]

If the old logs contains a specific pattern (for example a label named 'test' or same fields contains a test value) you can define an if condition at the top of the parser to drop {} all logs contains a specific pattern. 

[SoarMike]

Just wanted to touch on this again I managed to implement pattern dropping based on time for the logs we do not need but I am still getting one more validation error. The issue is with this one it does not even specify use log so I can test on it just curious if you may have any ideas?