How to get dataflow log by command line

[NguyenThanh]

Hi everyone

I have one question related to how to get log of stream-pipeline which is running in Dataflow by command line instead of UI. I have a certificate file to authenticate and work in Cloud service. But I do not have a username/password to log in UI interface. Therefore I cannot view Log created from Pipeline, I just have it right on command-line (using the certification to authenticate).

I try to some command such as  gcloud beta dataflow logs list <JOB_ID> --importance=detailed --verbosity=debug. But it does not show Log that I print in pipeline code as I expected. 

I have a question that Is there any way to get the log that I put in Pipeline by using command line or any others way?

Thank you!

[kolban]

My first thought is that logs written by Dataflow are written to the Google Cloud Logging subsystem.  We can find that documented here:

https://cloud.google.com/logging/docs

At the highest level, this is where all logs written by GCP products (including Dataflow) are sent.  Typically, customers log into GCP Console and view the logs through the GCP Console.  However, if you have no access to the GCP console but do have credentials for a GCP service account, one can use a gcloud command to access the logs from the command line:

https://cloud.google.com/sdk/gcloud/reference/logging/read

This command can use credentials supplied in a keys file for access.  However, we assume that the credentials for the service account themselves have authority to access the logs.  I would be tempted to suggest that you go visit with your company GCP admins and tell them what you are hoping to achieve.  There is no obvious reason that if you have a business need to look at logs and can do so with a credentials file that you have access to that they couldn't authorize your GCP Google Account to read logs (and perhaps just read logs and nothing more).  My experience with working with the raw data of logs is that it will work in a pinch but is no where nearly as elegant as working with logs in a UI.