Hi!
I have a Dataproc Single Node Cluster that is used as a Spark History Server.
However I notice in Cloud Logging that this is creating quite a lot of logs.
Looking at the documentation I can see that indeed there are a number of components that will generate logs: https://cloud.google.com/dataproc/docs/guides/logging#cluster_logs_in
But how do I alter/modify the behaviour of these logs? Ideally I want to set the log level to ERROR/OFF if possible.
Of course I could always create a Exclusion Rule on the Sink level but that feels like I'm fixing the problem at the wrong end.
Any ideas, since it isn't obvious from the documentation what is possible