LinkedIn
Twitter
Is there an example, or a recommended way to deduplicate messages that come from Cloud Storage Pubsub Notifications? I'm specifically interested in identifying duplicates of event type "Object Finalize" (https://cloud.google.com/storage/docs/pubsub-notifications#events), namely new objects being created.
I'm expecting the primary key to be a combination of fields like (id, generation), but not sure as of now. I found some useful info in this blog post titled Handling duplicate data in streaming pipelines using Dataflow and Pub/Sub but since I'm not using Dataflow or BigQuery I need to implement this myself. The couple of images I've posted from that blog point to the fact that I need to implement this in a custom way.
