Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

How to track incorrect layout of records in Log bucket?

tanu3012_0-1687059843573.png

This is a correct record as per layout. In our case textpayload is in jason format. How to handle any record whose textpayload is coming in any other format, for example text.

0 1 163
1 REPLY 1

Hi @tanu3012,

textPayload field is an unstructured logging and you can't index its content. You should see about structured logging. Please see this public documentation which discusses structured logging:

When the log payload is formatted as a JSON object and that object is stored in the jsonPayload field, the log entry is called a structured log. For these logs, you can construct queries that search specific JSON paths and you can index specific fields in the log payload. In contrast, when the log payload is formatted as a string and stored in the textPayload field, the log entry is unstructured. You can search the text field, but you can't index its content.


It is best to detect logs in JSON format. This documentation will help you decide if your application can generate structured logs.

Logging recognizes structured logs if you send the logs to the Logging API in the structured-logging format. Client libraries provide the methods for handling this format.

There are two way of writing structured logs: one is to set specific fields in the LogEntry envelope, and the other is to set the jsonPayload field within the LogEntry envelope. The schema for the former is determined by Cloud Logging, while the schema for the latter is determined by the user.

You must configure the agent to recognize structured logs. By default, the agents are configured to detect logs in JSON format and to handle them as structured logs. If your application has its own log format that you can't change, but you want the logs to be recognized as structured logs, then you must write logs in the structured-logging format, usually JSON, to stdout and stderr, so that the agents can recognize them as structured logs. Otherwise, you must configure your agent to understand your own format.


Here's how you can write structured logs.

Hope this helps.

 

Top Solution Authors