I am developing a custom transformation plugin using java for Google DataFusion, and I need to determine the total number of input records before starting the transformation process.
Since the transform(StructuredRecord input, Emitter emitter) method is executed for each individual record, I have the following questions:
How can I get the total count of input records before processing begins?
Is there a way to determine whether the current record being processed is the last one?
Is there any method that runs immediately after transform completes, which has access to the variables or data structures used within transform?
Any insights for handling these scenarios in a DataFusion transformation plugin would be greatly appreciated.
LinkedIn
Twitter
