Enable json logging for gmp components

[carstenthiel-te]

We are seeing a lot of info-level log messages from gke services classified as error.

Specifically prometheus in namespace gmp-system and gke-metrics-agent in kube-system are outputting logfmt strings that aren't parsed correctly. Is there a way to enable json log output for them?

[garisingh]

Would you mind posting a couple of examples of what you are seeing?

[carstenthiel-te]

Here is an example copied from Log Explorer

resource: {
labels: {
cluster_name: "**********"
container_name: "prometheus"
location: "europe-******"
namespace_name: "gmp-system"
pod_name: "collector-wqpc8"
project_id: "*************"
}
type: "k8s_container"
}
severity: "ERROR"
textPayload: "ts=2023-05-08T07:25:02.770Z caller=head.go:846 level=info component=tsdb msg="Head GC completed" duration=2.591371ms"
timestamp: "2023-05-08T07:25:02.806382218Z"