How to do Cloud Storage Validation

[jmaatheus]

In my storage I have several folders that represent tables in my database, they are fed via Kafka.

Within each main folder, there are subfolders representing the year, month and day.

I need a way to validate the last subfolder.

Is this possible through query?

Example:

SELECT FOLDER, MAX(SUBFOLDER)
  FROM STORAGE

[SebastianG]

I'm not sure that a by using a SQL like syntax would work - I'm joking, it won't, well almost, not sure if SteamPipe supports that / haven't tested.

But your problem can be solved via code, by that I mean using GCS APIs (https://cloud.google.com/storage/docs/reference/libraries) and check if that last subfolder exist or not. This would be the most elegant way to solve the problem.

Another one - bit more of a "hack" - is to make use of the gsutil and get the output from there. Not quite recommended.