Migrating old log messages after a structured log schema change

[grahamalama]

My team currently logs structured logs to STDOUT in a GKE environment. The structure of these logs doesn't take advantage of the special payload fields used by Google Cloud Logging. We're considering two options for how to fix that:

1. Change the structure of our logs emitted to STDOUT to include the special fields, e.g. use Google Cloud Logging severity levels instead of our custom severity level terms.

2. Do not change the structure of the logs emitted to STDOUT and instead use a custom Fluentbit Agent to parse our custom log format as something more Google Cloud Logging friendly.

In option one, we'd deploy our application with the changed log format, and in option two we'd deploy the custom logging agent.

In either case, what would happen with the old log messages? Would we have to "migrate" old messages in the log buckets to conform to the new schema with the special payload fields in order to see the benefits in the Logs Explorer? If so, how would that happen?

[RonEtch]

Hi  @grahamalama 

Welcome to Google Cloud Community!

In this case, either of your options will help you to conform and take advantage of the special payload fields. In relation to option 1, the same with your old log messages from the log bucket, you will need to restructure your application logs to the new log format. 

For you to be able to conform on the new log format and utilize the use of special payload fields you will need the following:

1. From old log messages, extract information that are relevant and map it to the log structure with the special payload fields.
2. Once the new logs are created from the said restructure, you can write it to the new or appropriate log buckets.

For references that are related to your query on how to manage your GKE logs, you may refer to the following links:

• Managing GKE logs
• Structured logging
• Configure the Logging agent
• Cloud Logging for GKE Workload
• Enabling Stackdriver Structured Logging on GKE

I hope this information is helpful.

If you need further assistance, you can always file a ticket on our support team.

[grahamalama]

Thanks @RonEtch. From the high-level process you described:

1. From old log messages, extract information that are relevant and map it to the log structure with the special payload fields.
2. Once the new logs are created from the said restructure, you can write it to the new or appropriate log buckets.

---

I found these two posts that seem like they'd allow me to complete this migration:

First export the logs to GCS: https://cloud.google.com/architecture/exporting-stackdriver-logging-for-compliance-requirements

Then, import the logs back into Cloud Logging: https://medium.com/google-cloud/importing-logs-from-cloud-storage-to-cloud-logging-ab4db28eada2

I'm guessing that we could perform our "migration" on the log files in the GCS bucket or in the Google Cloud Run import job.