Cases where datastream overwrites a file twice causing double triggering of cloud function

[RC1]

Hi folks , I have a usecase where the files ingested by datastream is used to trigger(object finalize)  cloud function for further processing. I have seen cases where datastream writes a file (with same name) twice (to GCS) causing double triggering of cloud function. Does datastream have such issues ?

[etaim · Accepted solution]

Hi, are you sure these are duplicate files with the same content? Datastream will, by design, write multiple files into the same folder in Cloud Storage if either (a) the file exceeds the limit set in the Datastream configuration (size and/or time), or (b) if Datastream detects a change in the schema (each file only contains events that correspond the same schema).