Converting Regular Airflow Logs to Structured JSON to write on Google Cloud Logging

[aakash-mahesha]

I am trying to solve a problem, where I can generate and store regular Airflow log files on a GCP VM. I am using google-cloud-ops-agent to write the logs directly from the VM to the cloud logging. I want the regular logs to be structured into JSONs in the order of 
{ dag_id: { run_id : { task_id : { attemp#: { payload_message}}}}}

I am trying to configure the google-cloud-ops-agent config file as follows :

logging:
  receivers:
    airflow_logs:
      type: files
      include_paths:
        - home/aakashmahesha/logs/dag_id=*/run_id=*/task_id=*/attempt=*.log
  processors:
    parse_json:
      type: parse_json
    extract_fields:
      type: parse_regex
      regex: '^/dag_id=(?P<dag_id>[^/]+)/run_id=(?P<run_id>[^/]+)/task_id=(?P<task_id>[^/]+)/attempt=(?P<attempt>\d+)\.log$'
      parse_from: attributes["agent.googleapis.com/log_file_path"]
  service:
    pipelines:
      airflow:
        receivers: [airflow_logs]
        processors: [parse_json, extract_fields]

 But when trying to restart google-cloud-ops-agent, it throws errors:

● google-cloud-ops-agent.service - Google Cloud Ops Agent

     Loaded: loaded (/lib/systemd/system/google-cloud-ops-agent.service; enabled; vendor preset: enabled)

     Active: failed (Result: exit-code) since Thu 2024-12-05 00:04:32 UTC; 7s ago

    Process: 582004 ExecStartPre=/opt/google-cloud-ops-agent/libexec/google_cloud_ops_agent_engine -in /etc/google-cloud-ops-agent/config.yaml (cod>




Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   41 |       type: parse_regex

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   42 |       regex: '^/dag_id=(?P<dag_id>[^/]+)/run_id=(?P<run_id>[^/]+)/t>

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]: > 43 |       parse_from: attributes["agent.googleapis.com/log_file_path"]

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:              ^

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   44 |   service:

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   45 |     pipelines:

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   46 |       airflow:

Dec 05 00:04:32 bigger-airlfow-vm systemd[1]: google-cloud-ops-agent.service: Control process exited, code=exited, status=1/FAILURE

Dec 05 00:04:32 bigger-airlfow-vm systemd[1]: google-cloud-ops-agent.service: Failed with result 'exit-code'.

Dec 05 00:04:32 bigger-airlfow-vm systemd[1]: Failed to start Google Cloud Ops Agent.

 

[ion_]

Hey @aakash-mahesha 

Do you have more logs available to you? I'm unfamiliar with everything involved but i'm happy to give a wrong answer and then maybe someone can correct both of us.

It looks to me like the error is pointing to the regex line, I would make that more simple (something like .*) and see if it still gives the same error?
Additionally, it might be that your forward slashes need to be escaped? I'm not sure if that is specific to the tools' syntax, but it looks like it might be an area of concern