Great responses. I'm not sure if you are a Google Cloud customer today or not. Either way, I am getting the impression that you might benefit from a focused and detailed design discussion for your specific use case. A Google customer engineer (technical) would be able to sit down with you (in person or virtually) and start gathering all your requirements and work with you to flesh our some high level architectures taking account of all the distinct needs. As for merging multiple streams of incoming events, especially if they need to be time windowed together is really starting to sound like Dataflow (Apache Beam). Not only can it group events into time windows for processing, it can also handle concepts such as late arriving events (such as might occur if one of the feeds broke or stalled). While Apache Beam SQL is indeed a candidate in a potential solution, I'd likely lean towards starting with Beam itself (Java or Python as opposed to SQL). The API mechanisms are more mature and richer at this time.
Again, we seem to be in a high level design discussion here and I suggest that the public forum won't be nearly as productive as engaging with a Google customer engineer (use that phrase when talking with Google ... they'll know what you mean). The Google customer engineer and yourself can likely quickly identify what parts of your puzzles can be easily solved by Beam (and Google's managed version ... Dataflow) and which parts might be trickier ... and ... if trickier ... bring to bear Google's experience working with other clients who had similar issues. If the puzzles get even trickier, leveraging the customer engineer would be the way to bring in additional Google subject matter experts who would be dedicated beam specialists (if needed).
LinkedIn
Twitter
