Detection rule won't match IOC domains uploaded to Entity Graph

[mmenegon]

Hey there,

I'm posting this hoping to find answers (and possibily a solution): In the last weeks I've been trying to upload IOC domains to SecOps using the method described here (https://medium.com/@thatsiemguy/ioc-matching-in-chronicle-siem-45a97c0b91a8) but I really can't get the POST request to work.

Allow me to elaborate. I'm posting to the endpoint "https://malachiteingestion-pa.googleapis.com/v2/entities:batchCreate" (documented here https://cloud.google.com/chronicle/docs/reference/ingestion-api#createentities) and passing a hostname-valued entity as the article suggested but although the API request is successful, I can't find the domain using a raw log search and our YARA-L rule won't match any log containing that domain.

Now, what got the domain uploaded to SecOps was to use a different structure. Instead of this:

"entity": {

                "hostname": "test.com"

        }

I've added the domain in a domain object while keeping the original hostname key (although this doesn't seem to change anything, the domain object is enough):

"entity": {

                "domain": {

                    "name": "test.com"

                },

                "hostname": "test.com"

        }

Now I can find test.com using a raw log search and see it being matched in the "Alerts & IOC" page. The real issue here is that our test rule (on which will be based all our production rules awaiting to be deployed ASAP) doesn't match any log.

I've already made sure that our rule is correct and all credentials have the right permissions. I'm sure we're talking about an issue related to the event structure but don't know which way to turn.

Would anyone be so kind to give me a hint on this? Your help would be greatly appreciated.

[dnehoda]

Hello, 

What does the rest of the log look like? Seems like something is missing that we would be expecting in order to ingest properly. 

[mmenegon]

Hello,

Here's the full json payload I'm uploading with sensitive data masked:

{
"customer_id": "CUSTOMER_ID",
"entities": [
{
"entity": {
"domain": {
"name": "test.com"
},
"hostname": "test.com"
},
"metadata": {
"collected_timestamp": "2024-09-26T06:52:55.013351+00:00",
"entity_type": "DOMAIN_NAME",
"interval": {
"end_time": "2024-09-27T07:17:55.013351+00:00",
"start_time": "2024-09-26T06:52:55.013351+00:00"
},
"product_entity_id": "PRODUCT_ENTITY_ID",
"product_name": "PRODUCT_NAME",
"threat": {
"threat_feed_name": "FEED_NAME",
"threat_id": "THREAT_ID"
},
"vendor_name": "VENDOR_NAME"
}
}
],
"log_type": "CATCH_ALL"
}

[James_E]

@mmenegon  Below is an example POST request specificaly for a DOMAIN_NAME entity_type:

body = {
    "customer_id": CUSTOMER_ID,
    "log_type": "UDM",
    "entities": [
        {
            "metadata": {
                "collected_timestamp": "2023-02-27T20:17:45.328295Z",
                "interval": {
                    "start_time": "2024-10-01T00:00:00Z",
                    "end_time": "2024-10-01T00:00:00Z"
                },
                "entity_type": "DOMAIN_NAME",
                "vendor_name": "TIP",
                "product_name": "ACME",
                "threat": {
                    "category_details": "C2",
                    "threat_id": "fdb7585c-34a2-474d-acf5-cc847834ee3d",
                    "threat_feed_name": "ACME-IOC-IP-DOMAIN"
                }
            },
            "entity": {
                "hostname": "logging.googleapis.com"
            }
        }
    ]
}

As outlined in the article you want to use entity.hostname as that UDM field is what's used for IOC matching. 

Once you've uploaded your entity, you'll see matches in Alerts & IOCs > IOC MATCHES.

[mmenegon]

That seems to have done the trick, Thanks!

However, my IP addresses aren't always matched in my rules. Do you know if there are any limits to the amount of IOC uploaded per requests or per entity (aside from the 1MB request size limit)? Reading the documentation I assumed that you could upload multiple IPs within the same "entity" object, like so:

{
    "customer_id": "CUSTOMER_ID",
    "entities": [
        {
            "entity": {
                "ip": [
                    "1.1.1.1",
                    "1.1.1.2",
                    "1.1.1.3",
					...
                ]
            },
            "metadata": {
                "collected_timestamp": "2024-09-27T14:24:52.835932+00:00",
                "entity_type": "IP_ADDRESS",
                "interval": {
                    "end_time": "2024-09-27T14:49:53.602439+00:00",
                    "start_time": "2024-09-27T14:24:54.025908+00:00"
                },
                "product_entity_id": "PRODUCT_ENTITY_ID",
                "product_name": "PRODUCT_NAME",
                "threat": {
                    "threat_feed_name": "FEED_NAME",
                    "threat_id": "THREAT_ID"
                },
                "vendor_name": "VENDOR_NAME"
            }
        },
        {
            "entity": {
                "ip": [
                    "1.1.1.4",
                    "1.1.1.5",
                    "1.1.1.6",
					...
                ]
            },
            "metadata": {
                "collected_timestamp": "2024-09-27T14:24:52.835932+00:00",
                "entity_type": "IP_ADDRESS",
                "interval": {
                    "end_time": "2024-09-27T14:49:53.602439+00:00",
                    "start_time": "2024-09-27T14:24:54.025908+00:00"
                },
                "product_entity_id": "PRODUCT_ENTITY_ID",
                "product_name": "PRODUCT_NAME",
                "threat": {
                    "threat_feed_name": "FEED_NAME",
                    "threat_id": "THREAT_ID"
                },
                "vendor_name": "VENDOR_NAME"
            }
        },
    ],
    "log_type": "UDM"
}

Is my assumption correct? Or should I upload only one IP per entity object?

Thank you

[James_E]

Yes you can have multiple IP addresses that correlate to a single entity, as 'ip' is a repeated field. In your detection rule you would want to make sure you're using the any/all to match.

[mmenegon]

How do I do that? This is my rule:

rule test_ioc_ip_matching {
  meta:
    description = "Test harness for matching IOCs via Entity Graph"

  events:
    $e.metadata.event_type = "NETWORK_CONNECTION"
    $e.principal.ip = $asset
    $e.target.ip = $ip
    
    $ioc.graph.metadata.entity_type = "IP_ADDRESS"
    $ioc.graph.metadata.vendor_name = "VENDOR_NAME"
    $ioc.graph.metadata.threat.threat_feed_name = "FEED_NAME"
    $ioc.graph.entity.ip = $ip

  match:
    $asset over 1m

  condition:
    $e and $ioc
}

[jstoner]

There is no reason you could not have multiple IPs for an entity, but when it comes to IOCs, I'm not sure how often we would expect to have a single entity have multiple IP addresses. An internal asset, of course we would, but I'm trying to come up with a great reason to have a watch list of IOCs from my threat intel rolled together.

The ANY/ALL can work as part of a line of criteria like ANY target.ip = "10.10.10.10" but it doesn't apply to a join statement or a reference list. That should not be issue from a matching perspective as I believe we roll through every IP in the event side to match to the other side, but perhaps I am missing something.

[mmenegon]

Thanks for your feedback.

Alright, I'm now uploading one IP address per event and still, my rule is not matching whereas the IOC MATCHES page is. What am I doing wrong?

[DanDye]

It may be that the entity's interval.start_time and interval.end_time are not encompassing your event's timestamp. The entity example you provided is only valid for 25 minutes after the collected_timestamp.