GCP logs are not properly ingested in Wazuh via GCP Pub/Sub Pull subscription

[Maryam]

GCP logs are being forwarded to Wazuh using GCP Pub/Sub Subscription. The required logs are being forwarded to pub/sub subscription using sinks. Wazuh configuration to pull logs also seems fine. Yet all the logs are not being sent to Wazuh and alot of logs are still in the backlog. What can be the possible reason and solution of this log reception issue?

Delivery latency health score (GCP):

 ack_latency :Value:0

 expired_ack_deadlines :Value:1

 nack_requests :Value:1

 seek_requests :Value:1

 utilization :Value:0

[kolban]

If I 'm understanding properly, you are relaying Google Cloud Logs to the open source Wazuh product using Google Cloud Pub/Sub.  However, you are sensing that there are log messages that are not being ingested by Wazuh.  When you say they are in the "backlog" does that mean that they remain in Google Cloud Pub/Sub topics without having been consumed by a subscription?

Pub/Sub is a technical service that allows one component to write messages to them (eg. Cloud Logging) and another component (eg. Wazuh) to read from them.  If messages remain in the topic when you expect them to have been read, my immediate thought is that there is something amiss with the consumer (Wazuh).  Sadly, I have no experience/skills in Wazuh so can't offer any guidance.  I think our first plan of attack is to determine what is amiss and who is to blame.  Is it that Google Cloud Pub/Sub isn't presenting them to Wazuh correctly or is Wazuh not consuming them correctly.  There appears to be a Wazuh community site here.  I'd suggest posting a question there.  If we can determine that it is Wazuh vs Cloud Pub/Sub that would go a long way.  I'd also suggest looking in Cloud Logging itself to see if there are any errors or issues being logged there that might come from Wazuh itself.

[Maryam]

---

@kolban wrote:When you say they are in the "backlog" does that mean that they remain in Google Cloud Pub/Sub topics without having been consumed by a subscription?

---

Yes the consumed messages are not configured to be stored in backlog. Only unconsumed messages are stored in backlog.