Intermittent packet loss between cloud functions / PubSub / BigQuery

[analyticsdept]

Hi everyone,

Hoping someone knows something. I have a Function that receives data and writes it to a PubSub Topic. That topic then triggers another Function that posts it to BigQuery. There's also an App Engine service that also receives data directly and posts to PubSub / BigQuery (though this process is managed by Dataflow)... so, lots of moving parts but two distinct paths for data to travel.

Across multiple GCP projects, since about the 12th of March, I've seen significant loss of data making its way from end to end. I've yet to find one single point of failure, and I've gone through Cloud Status and all of the posted incidents but can't really find anything concrete in terms of a specific issue. Worth mentioning that all of my resources are in europe-west1 / Belgium.

Does anyone know of any unreported issues / solutions?
Thanks in advance!

[dikaur]

Hello,

There is no recent issue reported regarding the packet loss between Cloud functions / PubSub / BigQuery. 

When publishing to Pub/Sub via Cloud Functions, publish can happen asynchronously as one calls publish, but then doesn't wait for the result. This publish happens asynchronously. When a function's code runs successfully, then any remaining async work may be run at a much lower priority, resulting in timeouts in publishing to Pub/Sub and ultimately, the messages being dropped. If you are still experiencing the issue, you can :

• verify the "-retry" property set on their Cloud Functions (Cloud Function > Trigger > Retry on failure )
• view the  logs for the messages that are having issues to get more details
• view the Cloud Function logs  to confirm the/deny delivery of the message: