Hello,
Im working with the dataflow streaming flex templates documented here: https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-proto-to-bigquery
Our protobuf schema utilizes the struct type provided here:
https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/struct.proto
When I compile the .proto file into .pb using:
protoc -I=./ actions.proto --include_imports --descriptor_set_out=actions.pb
and attempt to start the job I get the following error from dataflow:
com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.
java.lang.IllegalArgumentException: Cannot infer schema with a circular reference. Proto Field: google.protobuf.Struct
at org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getSchema(ProtoSchemaTranslator.java:174)
In our case, the use of the struct type looks like this:
message Track {
// Name of the action that the user has performed
string event = 1;
// Free-form dictionary of event properties
google.protobuf.Struct properties = 2;
// Free-form dictionary of user properties
google.protobuf.Struct user_properties = 3;
}
From what I can tell, the error is due to Structs being able to contain other structs, is this just not supported by Java? is there a workaround to using the google provided Struct type? any experience or help would be greatly appreciated!
LinkedIn
Twitter
