This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Announcements
Webinar May 14th: Gemini's generative AI within Google SecOps
Webinar May 29th: Log Ingestion: Bindplane + Google SecOps
Log in to ask a question

Guide to using Ingestion Scripts with Google Security Operations

Posted on 11-06-2024 07:24 AM
and filed in security-blog
sumitpatel
Staff

Google Security Operations provides a set of ingestion scripts, written in Python, that can simplify security data ingestion. Using Cloud Run functions, you can deploy them and streamline your log collection process. Cloud Run is a service offered by Google Cloud that lets you run containers in a fully managed serverless environment, enabling you to deploy and scale without having to worry about managing servers or infrastructure. 

Currently supported log types are listed here in our public docs https://cloud.google.com/chronicle/docs/ingestion/ingest-using-cloud-functions and GitHub repository for the scripts https://github.com/chronicle 

In our scenario, we will be ingesting OneLogin user context (ONELOGIN_USER_CONTEXT) logs into our Google Security Operations instance. 

Before you begin

Read the following resources that provide context and background information that enable you to use the Google Security Operations ingestion scripts effectively.

Each sub-directory in Google Security Operations GitHub contains files that ingest data for a single Google Security Operations log type. The script connects to a single source device and then sends raw logs to Google Security Operations using the Ingestion API. We recommend that you deploy each log type as a separate Cloud Run function. You can access the scripts in the Google Security Operations GitHub repository. Each sub-directory in GitHub contains the following files specific to the log type it ingests.

  • main.py is the ingestion script specific to the log type. It connects to the source device and ingests data to Google Security Operations.
  • .env.yml stores configuration required by the Python script and is specific to the deployment. You can modify this file to set configuration parameters required by the ingestion script.
  • README.md provides information about configuration parameters.
  • Requirements.txt defines the dependencies required by the ingestion script. In addition, the common folder contains utility functions that all ingestion scripts depend on.

Lets Begin

1. Launch your IDE of choice, remember to install the Google Cloud CLI as detailed above. I will be using the Google Cloud Shell service for this blog.
Note: You can learn more about Google Cloud Shell here https://cloud.google.com/shell 

2. In Google Cloud, a Service Account is a special kind of account typically used by an application or workload, such as a Cloud Run Function, rather than a person. A service account is identified by its email address, which is unique to the account. For our solution, create a dedicated service account to run the OneLogin Cloud Function.

 

 

SA_NAME="sa-onelogin-to-chronicle"
SA_DESCRIPTION="Onelogin to Chronicle SIEM SA"
SA_DISPLAY_NAME="Onelogin Service Account"
gcloud iam service-accounts create $SA_NAME --description="$SA_DESCRIPTION" --display-name="$SA_DISPLAY_NAME"

 

 

3. Grant the required IAM role to your newly created Service Account

 

 

PROJECT_ID="[PROJECTID]"
gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com"  --role='roles/cloudfunctions.developer'

 

 

4. Use Google Cloud’s Secret Manager to store your ingestion Service Account credentials required in step 6.

 

 

SECRET_ID="secops_ingestion_SA"
CRED_PATH="[path_to]/[creds].json"
gcloud secrets create $SECRET_ID --replication-policy="automatic"
gcloud secrets versions add projects/$PROJECT_ID/secrets/$SECRET_ID --data-file=$[CRED_PATH]
gcloud secrets add-iam-policy-binding $SECRET_ID --member="serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --role='roles/secretmanager.secretAccessor'

 

 

After the secrets are created in Secret Manager, use the secret resource name as the value for environment variables. For example: projects/{project_id}/secrets/{secret_id}/versions/{version_id}, where {project_id}, {secret_id}, and {version_id} are specific to your environment.

You can repeat this above step to store the credentials for the security vendor. In my case I’ll run it again to store my OneLogin User Context API credentials 

 

 

SECRET_ID="onelogin_ingestion_SA"
CRED_PATH="/home/admin_/onelogincreds.json"
gcloud secrets create $SECRET_ID --replication-policy="automatic"
gcloud secrets versions add projects/$PROJECT_ID/secrets/$SECRET_ID --data-file=$CRED_PATH
gcloud secrets add-iam-policy-binding $SECRET_ID --member="serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --role='roles/secretmanager.secretAccessor'

 

 

5. Clone the GitHub repository and ensure we have the contents of the script folder you’re interested in. Ensure to include the common folder. 

I have created the onelogin_user scripts as below:

 

 

git clone https://github.com/chronicle/ingestion-scripts.git
cd ingestion-scripts
cd onelogin_user
cp -r ../common/ .

 

 

6. Edit the .env.yml file for the function and populate the required environment variables. The following table lists runtime environment variables common to all ingestion scripts.

Variable Name Description Required Default Secret
CHRONICLE_CUSTOMER_ID Google Security Operations customer ID. Yes None No
CHRONICLE_REGION
 Google Security Operations region. Yes us
Other valid values: asia-northeast1, asia-south1, asia-southeast1, australia-southeast1, europe, europe-west2, europe-west3, europe-west6, europe-west12, me-central1, me-central2, me-west1, and northamerica-northeast2.		 No
CHRONICLE_SERVICE_ACCOUNT
 Contents of the Google Security Operations service account JSON file. Yes None Yes
CHRONICLE_NAMESPACE
 The namespace that the Google Security Operations logs are labeled with. For information about Google Security Operations namespaces, see Work with asset namespaces. No None  

As an example my .env.yml file for OneLogin environment variables is:

 

 

# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
CHRONICLE_CUSTOMER_ID: xxxxx-xxxx-xxx-xxx-xxxxxx
CHRONICLE_REGION: australia-southeast1
CHRONICLE_SERVICE_ACCOUNT: projects/xxxxxxxx/secrets/secops_ingestion_SA/versions/1
CHRONICLE_NAMESPACE: ""
CLIENT_ID: xxxxxxxxxxxxxxxxxxxxxxxxxx
CLIENT_SECRET: projects/xxxxxxxxxx/secrets/onelogin_ingestion_SA/versions/1
TOKEN_ENDPOINT: https://api.us.onelogin.com/auth/oauth2/v2/token
# Keeping the default value as 30 minutes considering the frequency of OneLogin Users data.
POLL_INTERVAL: "30"

 

 

Hint: An easy way to make changes is to use “Cloud Shell Editor”. 

Lets save the .env.yml file

7. Now we can deploy the Cloud Function

 

 

gcloud functions deploy cf-onelogin-to-chronicle --gen2 --region=australia-southeast1   --project=$PROJECT_ID --runtime=python312 --entry-point=main --memory="4096MB" --trigger-http --env-vars-file .env.yml --service-account="$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --build-service-account=projects/$PROJECT_ID/serviceAccounts/$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com

 

 

8. Lets schedule the Cloud Function to run at regular intervals. The Cloud Schedule cron job should match the POLL_INTERVAL as configured in your .env.yml., e.g., 30 minutes by default.

 

 

LOCATION="australia-southeast1"
JOB_NAME="onelogin-to-chronicle"

gcloud scheduler jobs create http $JOB_NAME --location $LOCATION --schedule "*/30 * * * *" --time-zone "Etc/UTC" --uri  "https://$LOCATION-$PROJECT_ID.cloudfunctions.net/cf-onelogin-to-chronicle" --http-method POST --message-body "}" --headers="Content-Type=application/json" --oidc-service-account-email $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com

 

 

9. You can jump into Cloud Scheduler and “Force run” the job to kick it off. Once it has completed, we can see logs flowing through from OneLogin.

image.png

Summary

This concludes our guide to streamlining log ingestion into Google Security Operations using Cloud Run functions and readily available Python scripts. By following these steps, you can automate the ingestion of logs, enhancing your security monitoring capabilities. Remember to explore the diverse range of supported log types and adapt this process to suit your specific security data needs. This approach not only simplifies log management but also strengthens your organization's security posture by ensuring efficient and timely data analysis within Google Security Operations.

To deepen your understanding and explore further possibilities with Google Security Operations and Cloud Run functions, we recommend checking out the following resources:

7 2 29.3K
Authors
2 Comments