Enrichment with delayed data freshness reduces data quality

[JensW]

We use rapid7 insight feed (asset enrichment) to enrich our entity data model. The feed seems to run once a day. But this is an issue because rapid7 also enriches the ip (entity.asset.ip). In an environment where static ip addresses are not used, the mapping from ip address to hostname is quickly outdated. This results in enrichment with false hostnames (because of the outdated rapid7 data, that is used to enrich events).

What is the best way to deal with the issue?

Two ideas that came to my mind are to change the ingest schedule of the rapid7 feed, so that more recent data is available. This will shift the issue to rapid7 (where the data freshness is 6 hours). Or I could customize the parser and drop the ip addresses. Then I guess the mapping will only run via mac addresses.

Has anyone experienced similar issues  and how did you solve them?

[dnehoda]

It seems to me that it pulls data every minute.  

https://cloud.google.com/chronicle/docs/reference/feed-management-api#rapid7-insight

Also, just curious what the DHCP leases are set to.  Seems they expire too quickly and bounce around too often.  

[JensW]

@dnehoda Thanks for the reply. Well, our ingest schedule doesn't look like it pulls every minute data.

This also confirmed when I search for a specific asset hostname, I only see asset updates that are daily. Perhaps the every minute ingest schedule is for the endpoint vulnerabilities?

I think the DHCP leases are fine. The issue is mainly when clients switch from wired network to wifi (e.g. switching from workplace to meeting room). In this case, the IP changes independently of the lease time.

[dnehoda]

Looks like our docs are wrong.  

I would modify the time frame on the rapid 7 side first. 

if that doesn’t work we can modify the feed via command line with the proper permissions. 

[JensW]

Unfortunately, it is not possible to change the collection interval in rapid7. This means that rapid7 is the bottleneck and even modifying the feed would not change the data freshness (based on the dependency on rapid7).

This is why i have now changed the parser. The IP is now only parsed if the mac and hostname are empty. This should solve the issue.

[dnehoda · Accepted solution]

I thought above you said you could change it to 6 hours.   Modifying the parser works but that's not ideal - so maybe we could modify the gopher feed interval time in the command line for SecOps.   

[JensW]

The interval cannot be changed for rapid7, it is set to 6 hours by default. (https://docs.rapid7.com/insight-agent/data-collected/#data-collection)

But the 6 hours collection inverval from rapid7 might be "fresh" enough. We should try that out. Should I create a support ticket for changing the feed interval time?

[dnehoda]

I would believe that would be the best path forward.