Solved: GCP Batch no cloud logging when task is running - Page 2

gradientopt · 10-17-2023 06:24 PM

Hi, when I run GCP batch jobs with cloud logging, there is only logging available when a task stopped running (either failed or succeeded), how can I see the log as the task is running? This would be very helpful since

1.it allows us to monitor the running of tasks

2.if a task fails with exit status 137 (memory usage issue), all intermediate logging will be lost. If we can see the intermediate logs as the task is running, this can be prevented.

Thanks!

bobtian

Looks like there is some buffering mechanism in python. By default, Python buffers its stdout. When the buffer fills up or when the program finishes (and the buffer is flushed), you'll see the output.

To work around I can think of the following options

Option 1: add environment variable ”PYTHONUNBUFFERED=1“ to the Batch job spec like the following example

{
  "taskGroups": [
    {
      "taskSpec": {
        "environment" : {
          "variables" : {
             "PYTHONUNBUFFERED" : "1"
          }
       },
        "runnables": [
          {
            "container": {
              "imageUri": "YOUR_IMAGE_URL",
              "commands": ["/bin/sh", "-c","/execute.py"]
            }
          }
        ]
      }
    }
  ],
  "logsPolicy": {
    "destination": "CLOUD_LOGGING"
  }
}

Option 2 Run the python script in unbuffered mode using the `-u` option. e.g. if you are running using python3 you can do.

"runnables": [
     {
        "container": {
           "imageUri": "YOUR_IMAGE_URL",
           "commands": ["/bin/sh", "-c", "python3", "-u", "/execute.py"]
         }
      }
]

Option 3 set the environment in your Dockerfile, but this requires much effort since you may need to build image again.

ENV PYTHONUNBUFFERED 1

View solution in original post