Batch - structured logging

[adamreid]

Hi,

I have a long running R process in a container within a Batch Job.

The R process is writing to stdOut and I've set the Batch 'logsPolicy' destination to CLOUD_LOGGING  -  I correctly see the log output from the container in Cloud Logs.   However, Cloud Logs is interpreting my log output as text and not the structured logs we require.  Also, the severity of the log gets ignored. ie -  they are all INFO regardless

Is there any way to get structured logs into Cloud Logging using this setup? 

[aegolden]

Currently Batch interprets all outputs to STDOUT and STDERR as plain text. We create a LogEntry with textPayload equal to the output. The severity of the LogEntry is INFO for all entries generated by STDOUT, and the severity is ERROR for all entries generated by STDERR.

Can you give us an example of the output your process produces and the LogEntry (or multiple entries) you expect to result from that output? It sounds like maybe you are writing a JSON blob, or similar structured object, and want Batch to interpret that blob as a complete LogEntry, to be relayed to CloudLogging. Is that right?

[kolban]

I got to thinking about this puzzle.  Assuming that your application ("R") in this case is running in a container ... I'm tempted to approach it as follows:

Redirect your applications output to Linux syslog (eg. using the logger command)

Install and configure the Google fluentd in your image such that messages written to a specific syslog destination are written to Cloud Logging ... see Logging agent configuration

My thinking works as follows.  If we redirect your apps output to syslog then you maintain a vanilla (non-Google) oriented application that can still run anywhere.  By installing the Google Cloud Logging agent and configuring it properly, the Cloud Logging agent starts monitoring your syslog logs.  As your app writes new log entries into syslog, these get relayed into Cloud Logging.  However, because we are now using agent logging, we appear to have all the power of agent logging at our disposal.  One of the apparent flags/features of agent logging is the configuration flag called "detect_json".  It is documented as follows:

Whether to try to detect if the log record is a text log entry with JSON content that needs to be parsed. If this option is true, and an unstructured (text) log entry is detected as in JSON format, then it is parsed and sent as a structured (JSON) payload.

What this tells me is that if a "plain text" record is written to syslog, it will be relayed to Cloud logging as a text record while if the record written to syslog is detected to be JSON it will be logged as a JSON structured payload to Cloud Logging which is what I sense you want.  This technique sounds like it requires some setup but in the end you will have more flexibility as you will be able to control the Cloud Logging output with more fidelity.  You are effectively logging without going through Cloud Batch.

[adamreid]

Thanks for info @aegolden . 

---

It sounds like maybe you are writing a JSON blob, or similar structured object, and want Batch to interpret that blob as a complete LogEntry, to be relayed to CloudLogging. Is that right?

---

You are exactly right. It sounds like Batch does not support this currently. Are there any plans to introduce this? 

[Shamel]

Support for structured logging was added in Batch. You can find more details here.

[windymelt]

Hi, I'm running Batch job that emits structured log with label:

{"message": "hello", "logging.googleapis.com/labels": {"label1": "labelValue"}}

However it seems ignoring logging.googleapis.com/labels field.

Expected: label field in LogEntry contains label1 (merged to labels that is emitted from batch system).

Actual: jsonPayload."logging.googleapis.com/labels" field contains label1.

Does structured logging in Batch support labeling feature?

[bobtian]

Hello windymelt,

Hello windymelt, thanks for reporting, currently "logging.googleapis.com/labels" is not lifted from jsonPayload to the Cloud Logging entry.label. We are having a patch to fix this. This is expected to release in about 2 weeks.

As a workaround, are you using labels to analyze logging? For now, could you use "logging.googleapis.com/labels" to do some filter directly?

[windymelt]

Thank you for response and answer! I'll use "logging.googleapis.com/labels" until the patch rolls out.

[bobtian]

It's now supported. logging.googleapis.com/labels will now be shown in labels field.