Cloud Composer - No Logs in Cloud Logging

[lucas_rosa_dotz]

Hello,

I'm using Cloud Composer 2 and I've noticed that practically no logs seem to be flowing to Cloud Logging.

For example, the following is my log for a PythonOperator when saving the logs to the environment bucket:

If I search around this time in Cloud Logging, I can't find any of these messages. This worries me since if I were to move to Composer 3, the only option available is to log to Cloud Logging. It feels like only messages logged as Warning and above are reaching Logging, but I'm not sure. Since this is a default operator, I'm ruling out problems with custom operators.

Anyone noticed anything similar?

[caryna · Accepted solution]

Hi @lucas_rosa_dotz, 

Welcome to Google Cloud Community!

Here are the possible solutions to resolve your issue:

• Check your Cloud Composer 2 versions to help you understand the process of storing your task logs both in environment and Cloud Logging.
• Verify if your _Default logging bucket is enabled.
• Check your inclusion/exclusion filters on _Default Logging bucket. You may also find valuable insights in this related discussion on StackOverflow.
• Ensure that you have the necessary permissions to access Cloud Logging.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[lucas_rosa_dotz]

You're correct, I inherited this project and completely forgot to check the Logging. There were several exclusion filters turned on, including one for Composer. After disabling them I started seeing all logs normally.

Thanks!