OBJECT_FINALIZE multiple events when file uploaded to Cloud Storage

[soerendam-rm]

I have a client that uploads files to Cloud Storage. I have added a Pub/Sub Notification on OBJECT_FINALIZE. When I test it and upload the files myself via the GUI or via gcloud/gsutil exactly one notification is sent upon upload. However, when my client uploads the file to Cloud Storage multiple notifications are sent and my Cloud Function runs 100s of times. I suppose my client is streaming the upload and that causes the issues. Is there a way that I can get only one notification when the file is completely uploaded? Or would I need to look at the metadata in some way? 

[imtaher]

Yes, you're correct that the issue is most likely due to the streaming upload process used by your client. When a file is uploaded via streaming, the Cloud Storage service sends multiple OBJECT_FINALIZE events as the upload progresses.

To ensure that you receive only one notification when the file is completely uploaded, you can add some additional logic to your Cloud Function. One possible solution is to check the metadata of the uploaded file to see if it has already been processed. You can add a unique identifier to the metadata of the file before it is uploaded, and then check for the presence of this identifier in your Cloud Function.

For example, you could add a custom metadata field to the file called "processed" with a value of "false" when the file is uploaded. In your Cloud Function, you can check if the "processed" field is set to "false" before processing the file. Once the file has been processed, you can update the metadata of the file to set the "processed" field to "true" to indicate that it has already been processed.

Another option is to use a Cloud Storage bucket that is configured for event-based processing with Cloud Functions. This allows you to define a Cloud Function that is triggered only when a file is completely uploaded. To set this up, you can create a new Cloud Storage bucket and configure it to trigger a Cloud Function when a new object is created. In the Cloud Function, you can check if the object is fully uploaded before processing it.

I hope this helps! Let me know if you have any further questions. If want to get direct solution to your problems than you can ask questions in Kiask.xyz