Anthos structured logging not working as expected

[ivan-aracki]

I have the same application deployed in GKE and in Anthos AWS cluster.

Once I open Logs Explorer, I can see that only the apps from the Anthos cluster are having all the logs within `jsonPayload.message` as one json string. It's not passing the logs as it should be according to https://cloud.google.com/logging/docs/structured-logging.

Any idea, what could be the problem? Could it be that default fluentbit config has wrong Parsers within Anthos cluster?

[ivan-aracki]

Easiest way to test this would be:

kubectl run test-logging-app --quiet --restart=Never --image=alpine -- sh -c 'sleep 3; echo "{\"message\": \"Hello from Kubernetes!\", \"severity\": \"WARN\"}"'

In the Anthos cluster I don't get WARNING severity although I am getting that in the classic GKE cluster.

[ivan-aracki · Accepted solution]

After many tweaks of the configuration, it seems we had to split parser [FILTER] for k8s_application*:

    [FILTER]
        Name         parser
        Match        k8s_application*
        Key_Name     message
        Reserve_Data True
        Parser       cri
        Parser       appglog
        Parser       json

to

    [FILTER]
        Name         parser
        Match        k8s_application*
        Key_Name     message
        Reserve_Data True
        Parser       cri
    [FILTER]
        Name         parser
        Match        k8s_application*
        Key_Name     message
        Reserve_Data True
        Parser       appglog
        Parser       json

That way appglog and json parsers started to work.