Converting Regular Airflow Logs to Structured JSON to write on Google Cloud Logging

[aakash-mahesha]

I am trying to solve a problem, where I can generate and store regular Airflow log files on a GCP VM. I am using google-cloud-ops-agent to write the logs directly from the VM to the cloud logging. I want the regular logs to be structured into JSONs in the order of 
{ dag_id: { run_id : { task_id : { attemp#: { payload_message}}}}}

I am trying to configure the google-cloud-ops-agent config file as follows :

logging:
  receivers:
    airflow_logs:
      type: files
      include_paths:
        - home/aakashmahesha/logs/dag_id=*/run_id=*/task_id=*/attempt=*.log
  processors:
    parse_json:
      type: parse_json
    extract_fields:
      type: parse_regex
      regex: '^/dag_id=(?P<dag_id>[^/]+)/run_id=(?P<run_id>[^/]+)/task_id=(?P<task_id>[^/]+)/attempt=(?P<attempt>\d+)\.log$'
      parse_from: attributes["agent.googleapis.com/log_file_path"]
  service:
    pipelines:
      airflow:
        receivers: [airflow_logs]
        processors: [parse_json, extract_fields]

 But when trying to restart google-cloud-ops-agent, it throws errors:

● google-cloud-ops-agent.service - Google Cloud Ops Agent

     Loaded: loaded (/lib/systemd/system/google-cloud-ops-agent.service; enabled; vendor preset: enabled)

     Active: failed (Result: exit-code) since Thu 2024-12-05 00:04:32 UTC; 7s ago

    Process: 582004 ExecStartPre=/opt/google-cloud-ops-agent/libexec/google_cloud_ops_agent_engine -in /etc/google-cloud-ops-agent/config.yaml (cod>




Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   41 |       type: parse_regex

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   42 |       regex: '^/dag_id=(?P<dag_id>[^/]+)/run_id=(?P<run_id>[^/]+)/t>

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]: > 43 |       parse_from: attributes["agent.googleapis.com/log_file_path"]

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:              ^

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   44 |   service:

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   45 |     pipelines:

Dec 05 00:04:32 bigger-airlfow-vm google_cloud_ops_agent_engine[582004]:   46 |       airflow:

Dec 05 00:04:32 bigger-airlfow-vm systemd[1]: google-cloud-ops-agent.service: Control process exited, code=exited, status=1/FAILURE

Dec 05 00:04:32 bigger-airlfow-vm systemd[1]: google-cloud-ops-agent.service: Failed with result 'exit-code'.

Dec 05 00:04:32 bigger-airlfow-vm systemd[1]: Failed to start Google Cloud Ops Agent.