BigQuery Subscription: Incompatible Schema using proto3 and BigQuery Subscriber

[pzwikirsch-uc]

Hi all,

it seems like there is an issue/bug using proto3 for the Cloud Pub/Sub Topic Schema definition together with BigQuery Subscriptions.

I defined the following proto3 schema for my Pub/Sub topic:

syntax = "proto3";
message MyMessage {
  int64 ts = 1;
  string someId = 2;
}

The schema of my BigQuery table is defined as follows:

[
  {
    "name": "ts",
    "type": "INTEGER",
    "mode": "REQUIRED"
  },
  {
    "name": "someId",
    "type": "STRING",
    "mode": "REQUIRED"
  }
]

When I try to create a BigQuery Subscriber, the system always responds that the schemas are incompatible due to the fields in the table being required, but nullable in the topic.

The error:

│ Error: Error creating Subscription: googleapi: Error 400: Incompatible schema: field someId is required in table, but nullable in topic.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "pubsub.googleapis.com",
│     "metadata": {
│       "actual_value": "REQUIRED",
│       "expected_value": "NULLABLE",
│       "field_name": "someId",
│       "reason": "INCOMPATIBLE_MODE"
│     },
│     "reason": "INCOMPATIBLE_SCHEMA"
│   }
│ ]

Even though that can't be. According to the official proto3 Default Values documentation, there are no NULL values in proto3 and per default, proto3 uses default values.

Is GCP aware of this issue and is there any estimate in terms of fixing this bug?

Thanks in advance.