google.cloud.logging_v2 not returning logs

[nick-youngblut]

My code:

def get_failed_jobs(project_id, max_jobs=5):
    # Initialize the Batch client
    batch_client = batch_v1.BatchServiceClient()

    # Define your project and location
    region = "us-west1"
    parent = f"projects/{project_id}/locations/{region}"
    client = batch_v1.BatchServiceClient()

    # Fetch the most recent batch jobs, filtering for failed ones
    failed_jobs = []
    for job in client.list_jobs(parent=parent):
        if str(job.status.state) == 'State.FAILED':
            failed_jobs.append(job)
        if len(failed_jobs) >= max_jobs:
            break
    return failed_jobs


def get_logs_for_job(job_id, project_id):
    # Initialize the Logging client
    log_client = logging_v2.Client(project=project_id)

    # create logger
    logger = log_client.logger("batch_task_logs")

    # set filter
    log_filter = f'resource.labels.job_id="{job_id}"'

    # get logs
    try:
        logs = logger.list_entries(filter_=log_filter)
        # get log entries
        log_entries = [entry.text_payload for entry in logs]
        return log_entries
    except Exception as e:
        print(f"Error fetching logs for {job_id}: {e}")
        return []


def main(args):
    # set GCP project
    project_id = "my-gcp-project"

    # get failed jobs
    failed_jobs = get_failed_jobs(project_id, max_jobs=5)

    # write logs
    for job in failed_jobs:
        logs = get_logs_for_job(job.uid, project_id)
        # Process or display logs
        print(f"Logs for job {job.uid}:")
        for log in logs:
            print(log)

The output:

Logs for job nf-0a509c4d-170269-76556b37-ax58-45990:
Logs for job nf-83f6c7c7-170268-545aaf76-15a6-4a990:
Logs for job nf-4a97b993-170269-fae4fb13-ed7a-441f0:
Logs for job nf-ef91af07-170268-8416f231-372a-44280:
Logs for job nf-2ea181f6-170268-684edcf5-41ae-43fa0:

I don't understand why the text for each log is not returned by `

[entry.text_payload for entry in logs]`. Gemini is terrible at answering such questions about the GCP API, and the GCP API docs are very hard to understand (dense and often rather unorganized). 

I should note that I'm using:

google-cloud-logging==3.8.0

google-cloud-storage==2.13.0

google-cloud-batch==0.17.6

Any suggestions would be appreciated!

[NoCommandLine]

1. I believe the output for each log in logger.list_entries(filter_=log_filter) should be a LogEntry
2. If so, looking at the documentation for LogEntry, I don't see text_payload as a parameter or property.
3. Try  calling the to_api_repr method i.e. 'entry.to_api_repr()'  (it should give a JSON format of the log entry) and see if that solves your problem

.... NoCommandLine ......

 https://nocommandline.com

A GUI for Google App Engine
    & Datastore Emulator

[nick-youngblut]

Thanks @NoCommandLine for the help! My problem is that `logger.list_entries(filter_=log_filter)` returns a generator, which is empty.  My log filter (`

f'resource.labels.job_id="{job_id}"'`) isn't working.

I'm essentially just using the example python code (https://cloud.google.com/batch/docs/analyze-job-using-logs)

from __future__ import annotations

from typing import NoReturn

from google.cloud import batch_v1
from google.cloud import logging


def print_job_logs(project_id: str, job: batch_v1.Job) -> NoReturn:
    """
    Prints the log messages created by given job.

    Args:
        project_id: name of the project hosting the job.
        job: the job which logs you want to print.
    """
    # Initialize client that will be used to send requests across threads. This
    # client only needs to be created once, and can be reused for multiple requests.
    log_client = logging.Client(project=project_id)
    logger = log_client.logger("batch_task_logs")

    for log_entry in logger.list_entries(filter_=f"labels.job_uid={job.uid}"):
        print(log_entry.payload)

 ...but it doesn't return any payload for any job uid. I just get empty generators. This is so frustrating.

[nick-youngblut]

Even if I just use `logger.list_entries()` without any filter, I just get an empty generator. I'm using a valid project id, so I don't see why I'm not getting ANY log entries.

[adaykin]

I'm having the same exact problem. The python API is returning no data even though I can see logs in the console explorer.