Deduplicating Email Addresses/Iterating over repeated fields in parser

Hey All

In the workspace user parser my users get the same email added to the entity.user.email_addresses field. 
sample user record with duplicated email addressessample user record with duplicated email addresses

This is slightly annoying, so I tried to remove it. My idea was to just dedupe at the bottom of the parser so that I can merge in future changes relatively easily.

However, i'm struggling to understand what my options are for dealing with an array/repeated field. It looks like i'm not able to iterate over, reference individual values or flatten it back to a string. The only thing I can find that works is checking for existence, so I can have a bunch of if statements that look something like this:

 

 

if [var_email] =~ /.*specificallyExcludedDomain\.com$/ or [var_email] in [var_entity][user][email_addresses]{
}
else {
  mutate{
    merge => {
      "var_entity.user.email_addresses" => "var_email"
    }
  }
}

 

 


Am I missing an obvious way to interact with these repeated fields in the parser? Any ideas would be awesome. 

0 1 212
1 REPLY 1