Logging service client returns a lot less log entries than what I got from log explorer

[waynez]

In Log Explorer, run the following query for the last 30 days got 51 entries.

resource.type="cloud_dataproc_cluster"

resource.labels.region="us-central1"

resource.labels.project_id="my-dataproc"

protoPayload.methodName="google.cloud.dataproc.v1.ClusterController.CreateCluster"

log_name="projects/unravel-dataproc/logs/cloudaudit.googleapis.com%2Factivity"


The Python client only got 8 log entries (protobuf entry) back, all with severity='ERROR'.

client = gc_logging.Client(project=project_id)

filter_expression = 'resource.type="cloud_dataproc_cluster"'

result_generator = client.list_entries(

filter_=filter_expression,

)
# only 8 entries got returned, serverity='ERROR'

[jangemmar]

Hi @waynez,

Welcome to Google Cloud Community!

It seems like you are experiencing inconsistencies in log retrieval between your Log Explorer and your Python client code using the Cloud Logging API.

In your Log Explorer query, It uses a very specific filter targeting CreateCluster operations in a particular region and project. Also, your python code uses a broader filter resource.type="cloud_dataproc_cluster" which will include all Dataproc cluster-related log entries, not just CreateCluster operations.

To isolate the issue, here’s the workaround that you can perform on your end:

• Make your python code’s filter as specific as your Log Explorer query
• Ensure your Python code explicitly sets the same 30-day time range using the start_time and end_time parameters in the list_entries method.
• Double-check the log retention policy in your Google Cloud project to ensure that logs from the past 30 days haven't been deleted.

You should be able to reconcile the log entries between your Log Explorer and python client by carefully aligning your filters and time ranges.

I hope the above information is helpful.