Ops Agent - Logging configuration updates not being ingested

[swhaley11]

I only see syslogs in the Log Explorer in GCP and not any of our custom logs.  Previously our company has been successfully using the Legacy Logging (fluentd) and specifying custom configurations.  We've installed the newer logging/Monitoring (Ops Agent) on several linux servers, but updating the config.yaml file as specified in the documentation doesn't seem to work:

https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/configuration#logging-config

Here is the format I'm using in the config.yaml file:

logging:

  receivers:

    customlogs:

      type: files

      include_paths:

      - /var/build/custom/logs/custom.log

I did restart the Ops Agent service after updating the configuration.  I also checked the Ops Agent logs and don't see any reference about these custom logs or any failures to import changes.  Has anyone had similar issues or is there an error I'm missing in my format?  Thanks!

[kylebenson]

Hey @swhaley11 ! Thanks for reaching out, sorry to hear you're having trouble here. 

I don't see anything wrong with this config, assuming it's in the right location and the permissions (or selinux on some systems) aren't problematic... are you seeing generic syslog logs appear in Log Explorer? 

If you didn't see any errors using:

$ sudo service google-cloud-ops-agent restart \
    && sudo service google-cloud-ops-agent status

For example:

$ sudo journalctl -xe | grep "google_cloud_ops_agent_engine"
Jun 30 22:00:26 centos8-2 google_cloud_ops_agent_engine[1141491]: 2021/06/30 22:00:26 the agent config file is not valid YAML. detailed error: yaml: line 21: did not find expected key

You can also check how the service is fairing with the following:

curl -s localhost:2020/api/v1/uptime | jq | grep uptime_sec

$ curl -s localhost:2020/api/v1/metrics | jq

$ curl -s localhost:2020/api/v1/metrics | jq

If you don't have `jq` installed, you can also format the output in a different tool for ease of reading. 

[swhaley11]

Yes the generic logs are being pulled and I can see them in the Logs Explorer.  I ran your commands anyway just to be sure.  The service is "active," the journalctl command didn't display any errors.  I believe the GCP permissions must be right, otherwise I wouldn't see syslogs.

[michaelmasouras]

We are facing a similar problem - but no logs at all from the VM. My custom configuration file:

logging:
  receivers:
    mhlogs:
      type: files
      include_paths: [/home/lw/log/*.log]

The service restarts without any complaints and is running, but not logs can be seen in the logs explorer. 

@kylebenson the curl commands to the api endpoint you mention return completely empty for us

[michaelmasouras · Accepted solution]

We figured out you need one entry for service/pipeline for it to work. 

service:

pipelines:

default_pipeline:

receivers:

- mhops

[swhaley11]

Thank you!!!  I finally got it to work using your suggestion:

logging:
  receivers:
    customlogs:
      type: files
      include_paths:
      -/var/build/custom/logs/*.log

  service:
    pipelines:
      default_pipeline:
        receivers:
        - customlogs

[Vikash2607]

hello ,

post installation of ops-agent my two services are remain disable  one is loggin-agent and second is metric-agent so what i need to configure in order to make them enabled ? please help 

[Vikash2607]

attached are the screnshots