Unable to create pub/sub BQ subscription using top...

-Mali · 02-07-2023 09:57 AM

Hello there-

I am trying to create a BQ subscription using a topic schema. I am using the console to create the subscription using my credentials as the owner of the project. I am getting an error message during "Request contains an invalid argument". I went through the permission related issues from another post and have the right permissions.

{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 3,
"message": "Request contains an invalid argument."
},
"authenticationInfo": {
"principalEmail": "*****@***.**",
"principalSubject": "user:*****@***.**""
},
"requestMetadata": {
"callerIp": "ipv6-address masked",
"callerSuppliedUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36,gzip(gfe)",
"requestAttributes": {
"time": "2023-02-06T16:44:27.317741907Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "pubsub.googleapis.com",
"methodName": "google.pubsub.v1.Subscriber.CreateSubscription",
"authorizationInfo": [
{
"resource": "projects/projectid-masked",
"permission": "pubsub.subscriptions.create",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/proj-id/subscriptions/sub-name",
"request": {
"messageRetentionDuration": "604800s",
"@type": "type.googleapis.com/google.pubsub.v1.Subscription",
"ackDeadlineSeconds": 10,
"retainAckedMessages": false,
"topic": "projects/proj-id-masked/topics/test",
"name": "projects/project-id-masked/subscriptions/test-sub",
"bigqueryConfig": {
"table": "proj-id-masked.new_env.test"
}
}
},
"insertId": "vwy0z3csss",
"resource": {
"type": "pubsub_subscription",
"labels": {
"subscription_id": "projects/project-id-masked/subscriptions/test-sub",
"project_id": "project-id-masked"
}
},
"timestamp": "2023-02-06T16:44:27.309009465Z",
"severity": "ERROR",
"logName": "projects/project-id-masked/logs/cloudaudit.googleapis.com%2Factivity",
"receiveTimestamp": "2023-02-06T16:44:28.454753646Z"
}

Not sure if this is a avro schema related issue for pub/sub. Here's the schema for the topic which matches BQ table schema.

[
{
"fields": [
{
"name": "field1",
"type": "STRING"
},
{
"name": "field2",
"type": "STRING"
},
{
"name": "field3",
"type": "STRING"
}
],
"name": "record1",
"type": "RECORD"
},
{
"fields": [
{
"name": "fied1_new",
"type": "STRING"
},
{
"name": "field2_new",
"type": "STRING"
},
{
"name": "field3_new",
"type": "STRING"
}
],
"name": "record2",
"type": "RECORD"
}
]

Here's topic schema

[
  {
    "fields": [
      {
        "name": "field1",
        "type": "string"
      },
      {
        "name": "field2",
        "type": "string"
      },
      {
        "name": "field3",
        "type": "string"
      }
    ],
    "name": "record1",
    "type": "record"
  },
  {
    "fields": [
      {
        "name": "fied1_new",
        "type": "string"
      },
      {
        "name": "field2_new",
        "type": "string"
      },
      {
        "name": "field3_new",
        "type": "string"
      }
    ],
    "name": "record2",
    "type": "record"
  }
]

Further analysis: If I uncheck the box "use topic schema" the subscriptions gets created without the 'data' field required in the BQ table. Looks like some kind of a bug.

Question to product team: Can we have a nested json in the topic or it must flat json object.

Thanks,

-Mali

nceniza

It appears that you can use nested approach for the json showed in the sample here in documentation. https://cloud.google.com/pubsub/docs/schemas#schema_types

To investigate further I would suggest creating a case for this so that an agent can review your logs and other details that might be helpful for resolving this inquiry.
https://cloud.google.com/contact

-Mali

What's this limitation in the schema type as per the link you provided?

For a schema to work with Pub/Sub, "only one top-level" type may be defined. Import statements to reference other types are also not supported.

Roderick

Hi @-Mali you might find more specifics on Schema quotas and limitations here. Let me know if you are still looking for a solution to your questions!

-Mali

I flattened the tables and seems to work. I tested posting some messages to the publisher and got a message has been accepted but unable to verify the data has been updated on BQ table. Table seems to be empty.

Unable to create pub/sub BQ subscription using topic schema