Cloud Logging to Pub/Sub Project Issues

[NFbdr]

Good morning,

I am having trouble getting admin.googleapis.com logs from the organization level to the pub/sub project level. At this point in time the only logs that are ingested from the org level to the project level are ouath2.googleapis.com logs and login.googleapis.com logs. I have followed several google knowledge base guides on how to configure a pubsub project, and thankfully they did allow me to start ingesting logs from Workspace to GCP; however, the logs are not making the next leap from the Organization level to the Pub/Sub level - any guidance?

Also, I am not able to get Drive Activity logs whatsoever (the appropriate API's are enabled as far as I know). Is this something that I will just have to do via API within Google Workspace? Or is there something that I am missing? My ultimate goal is to have all Google Workspace organization logs within a Pub/Sub project so I can then forward them elsewhere.

Thank you in advance. 

[ruthseki]

Hi @NFbdr,

Welcome to Google Cloud Community!

Getting all Google Workspace organization logs into a single Pub/Sub topic requires an approach involving careful sink configuration, proper IAM roles, and potentially integrating other APIs (like the Drive API) directly. Start by checking the permissions and sink configurations, and gradually test each component until the issue is identified. The lack of Drive Activity logs almost certainly points to needing the Google Workspace Drive API to access the data.

For your concern about admin.googleapis.com logs not flowing to Pub/Sub:

• Sink Configuration: Double-check your sink's settings. Ensure:
• Filter: The filter is correctly targeting logName: "projects/YOUR_PROJECT_ID/logs/admin.googleapis.com" (replace YOUR_PROJECT_ID with the ID of the organization's project you're targeting. This is crucial; you might be accidentally filtering for logs within your Pub/Sub project instead of the org level project). You might need a broader filter if you want all admin logs from all projects under the organization. Consider using a filter based on resource.type for more granular control.
• Destination: The destination is your Pub/Sub topic's fully qualified name (e.g., projects/your-pubsub-project/topics/your-topic).
• Permissions: Verify the service account associated with your sink has the necessary roles. The service account needs at least the roles/logging.logWriter role on the organization's projects and roles/pubsub.publisher on the Pub/Sub topic. Remember to propagate these roles appropriately up the hierarchy if necessary. This is critical - lacking appropriate org-level permissions is the most common reason for this failure.
• Sink Location: The sink's location needs to match (or be global) to both the source logs location and the PubSub topic location. If using regional resources this is crucial for data transfer.
• Organization vs. Project Logs: Understand that admin.googleapis.com logs are inherently generated at the project level, not necessarily the organization level. While you might see the logs aggregated in the organization's logs explorer, the actual log entries are associated with specific Google Cloud projects. Your sink needs to capture logs at that granular project level. You might need to create multiple sinks (or use a more sophisticated filter).
• Log Entry Types (Important): admin.googleapis.com logs are very diverse, and access to different parts may require different permissions, API access, or even specific Google Cloud services to be enabled. Carefully check the detailed documentation for admin.googleapis.com to identify whether any specific API's must be enabled on the individual projects within the organization and then check the permission roles.

Regarding your issue about Drive Activity logs not appearing:

• API Enablement (Revisit): Double-check if the Google Drive API is enabled at the organization level in the Google Workspace admin console, not just the GCP console. This is essential. There are separate APIs in Google Workspace itself that need to be enabled to get Drive Activity.
• Drive Activity API vs. Cloud Logging: Drive Activity logs are not directly ingested into Cloud Logging in the same way as other GCP logs. You will likely need to use the Drive API directly to retrieve Drive activity events. Cloud Logging isn't the primary mechanism for accessing them.

Here are the troubleshooting steps:

1. Verify Service Account Permissions:  Use the gcloud command-line tool or the GCP console to check the IAM permissions granted to the service account associated with your sinks.
2. Check Sink Errors: Go to Logging > Logs Explorer, and look for errors related to your sinks. This is where Google will usually report failures.
3. Simplify: Create a test sink targeting a small subset of logs (like a single project's admin.googleapis.com logs) to isolate whether the problem is with the organization-level access or with the sink configuration itself.
4. Examine Logs in the Pub/Sub Topic: After some time, check the messages arriving in your Pub/Sub topic. If nothing arrives, the problem is in the sink creation and not necessarily the Pub/Sub configuration.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[NFbdr]

Hi Ruthseki, 
I have all of the appropriate API's enabled; however, Drive Activity Logs are still not propagating in my GCP logs at the org level or the project level. What are some additional steps that you recommend? 

The goal is to have specific Google Drive Logs related to DLP policies to populate within GCP when the rules are triggered. Any guidance is much appreciated.