Using Bigtable connector in Application Integration to save the data in table but getting an error

[GayathriRam]

Hi Team,

I'm getting an error while saving the data to the table in bigtable.

The details of table are as follows:

gcloud bigtable tables describe idempotency_table --instance=idempotencytest --project=XXX
automatedBackupPolicy:
frequency: 86400s
retentionPeriod: 604800s
changeStreamConfig:
retentionPeriod: 604800s
columnFamilies:
idempotency_columns:
gcRule:
maxAge: 86400s
granularity: MILLIS  and in Application Integration the input for bigtable connector schema has {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"idempotency_columns": {
"type": "array",
"description": "DurationRule{maxAge\u003dPT24H}"
},
"RowKey": {
"type": "string"
}
}
} and I'm getting this error { "message": "{\"error\":{\"code\":500,\"details\":[{\"@type\":\"type.googleapis.com/google.rpc.ErrorInfo\",\"metadata\":{\"cause\":\"Cannot invoke \\\"com.google.protobuf.Value.getStringValue()\\\" because the return value of \\\"java.util.Map.get(Object)\\\" is null\",\"connection_type\":\"bigtable\",\"entity_type\":\"idempotencytest.idempotency_table\"}}],\"message\":\"Unable to create entity.\",\"status\":\"UNKNOWN\"}}", "code": 500 } and then the sample payload is{ "RowKey": "test-001", "idempotency_columns": [ { "status": { "value": { "string_value": "PENDING" } }, "retry_count": { "value": { "number_value": 2 } }, "is_retry": { "value": { "bool_value": true } } } ] }  . could you please help us with the sample payload to save the data in the table. 

[AlexET · Accepted solution]

Hey @GayathriRam, we’ve noticed your question hasn’t been answered yet, but we’ll keep it on our radar and reach out to others in the community to chime in.

[AlexET · Accepted solution]

Hey @GayathriRam, we’ve noticed your question hasn’t been answered yet, but we’ll keep it on our radar and reach out to others in the community to chime in.

[rabustamante · Accepted solution]

Hi @GayathriRam ,

The error Cannot invoke "com.google.protobuf.Value.getStringValue()" because the return value of "java.util.Map.get(Object)" is null indicates a mismatch between the JSON payload you are sending and the structure the Bigtable connector expects. The connector is looking for a specific field within your idempotency_columns array objects, but it can't find it because the payload is not formatted correctly.

The connector expects an array of objects for the column family, where each object explicitly defines a single column's qualifier and its value. Your current payload groups all columns into one object.

Correct Sample Payload

To fix this, you need to restructure your payload. Instead of one object with multiple keys in the array, create a separate object for each column you want to write.

Here is the correct sample payload to save the data:

JSON

{
  "RowKey": "test-001",
  "idempotency_columns": [
    {
      "qualifier": "status",
      "value": "PENDING"
    },
    {
      "qualifier": "retry_count",
      "value": 2
    },
    {
      "qualifier": "is_retry",
      "value": true
    }
  ]
}

Explanation of the Changes

1. Column Family (idempotency_columns😞 This key correctly maps to your column family and contains an array [].

2. Column Objects: Inside the array, each {} represents a single column to be written to the row.

3. qualifier Key: This is the required key the connector was looking for. Its value is the name of your column (e.g., "status", "retry_count").

4. value Key: This key holds the actual data for the corresponding qualifier. You can provide the value directly as a string, number, or boolean, and the connector will handle the data type conversion. You don't need the complex { "string_value": ... } structure.

This format allows the connector to correctly parse each column and its value before writing them to the Bigtable table under the specified row key.