How to make avro schema fields optional ?

[ankit-pradhan]

Already tried below schema.

{ 
"type": "record", 
"name": "MyRecord", 
"fields": [ 
{ "name": "requiredField", "type": "string" }, 
{ "name": "optionalField", "type": ["null", "string"], "default": null } ] }
{
"requiredField" : "test"
}

 

When I try to test message using Google Console UI test message against above schema with below message it gives error. 
Error : Message is invalid against schema.

Message tested : 
{
"requiredField" : "test"
}

[ms4446]

In Avro, to define a field as optional, you use a union type that includes "null". Your schema definition is correct. Here's how you define an optional field in Avro:

{
"type": "record",
"name": "MyRecord",
"fields": [
{ "name": "requiredField", "type": "string" },
{ "name": "optionalField", "type": ["null", "string"], "default": null }
]
}

The error you're experiencing might be due to how you're forming the message. When a field is optional in Avro and you want to omit it, you don't need to specify it at all in your message. If you want to set it to null, you must explicitly set it to null. The Avro specification treats the omission of a field and a field set to null as two different things.

Therefore, if you want to send a message without the optional field, you would do:

{
"requiredField" : "test"
}

And if you want to set the optional field to null, you would do:

{
"requiredField" : "test",
"optionalField": null
}

If you're still receiving an error, it's possible there might be an issue with how your message is being processed by the Google Console UI. Double-check the formatting and contents of your message to ensure there aren't any extra characters or missing brackets. If the issue persists, you may want to reach out to Google Cloud Support for further assistance.

[maximusg]

I think this must be an issue with schema validation on google pubsub. The schema ms4446 mentioned works fine on the avsc json library. and the exact same schema with the exact same test does not work with google pubsub. The only accepted value for the optionalField is null. even a string will not work. I've tested this both on the test message service and attached to a topic and leveraging JSON to send a string in. 

[hptortiz]

I'm having a hard time with this too.

The problem happens when a string is passed in a field that has the type: ["null", "string"].

I'm using your example @ms4446.

1. Using null value for optional field. OK

2. Using string in optional Field. FAILED!

This should be simple to accomplish. I'm doing something wrong?

[ms4446]

Your schema definition allows optionalField to be either null or a string, which is a correct implementation for optional fields in Avro. The problem arises when testing with a valid string value for optionalField in Google Cloud Pub/Sub, resulting in an error message that suggests a mismatch between the provided JSON message and the Avro schema.

Troubleshooting Guide for Avro Schema Issues in Google Cloud Pub/Sub

When encountering errors related to Avro schema compatibility or message validation in Google Cloud Pub/Sub, follow these steps to diagnose and potentially resolve the issue:

1. JSON Encoding for Avro Union Types

• Investigate the JSON encoding process. Ensure that the message conforms to standard JSON encoding practices for Avro, particularly for union types like ["null", "string"].

2. Schema Registration and Compatibility

• Verify Schema Registration: Confirm that the Avro schema registered in Google Cloud Pub/Sub matches the schema you're using for testing. Use the Google Cloud Console or gcloud CLI to retrieve and verify the schema.

• Check for Propagation Delays: Schema updates may not be immediately reflected due to caching or propagation delays. If you've recently updated the schema, wait a bit and then retry.

3. Tooling and Interface Limitations

• Console UI Quirks: Be aware that the Google Cloud Console UI might have specific formatting requirements or bugs. Test your schema and messages using alternative methods to rule this out.

• Alternative Testing Tools:

  • CLI Testing: Use the gcloud pubsub command-line tools to test publishing messages. This method bypasses the console UI.

  • Programmatic Testing: Publish messages using a Pub/Sub client library in your preferred programming language. This gives you precise control over message formatting and encoding.

4. Explicit Type Specification (Advanced)

• As a diagnostic step, try explicitly specifying the type in your message (e.g., {"string": "value"} for a string within a union). This isn't standard for JSON-encoded Avro and is recommended only to glean insights into the issue.

5. Information Gathering for Google Cloud Support

If the issue persists despite your troubleshooting efforts, prepare to contact Google Cloud Support with the following information:

• Exact Error Message: Document the precise error message(s) you're encountering.

• Schema Registration Details: Explain how and when the schema was registered, including any recent updates.

• Testing Methods Used: Detail the different methods and tools you've used for testing (UI, CLI, API).

• Troubleshooting Steps Taken: Summarize the troubleshooting steps you've already attempted.

[psvaiter-hdx]

I was struggling with that too and have just figured out why.

According to the Avro spec (URL Removed by Staff) when there's a union type, you have to specify the value as an object.

It should be as follows:

{
  "requiredField": "test",
  "optionalField": {
    "string": "teste"
  }
}

 However, it still doesn't work for me to omit the property in the payload, i.e. the field is not actually optional. Either I have to inform null or '{"string": "my value"}'. 🙁

Another tip: the Cloud Console does not show a useful error message. Use the CLI and you'll see a more descriptive error:

gcloud pubsub topics publish TOPIC_NAME \
  --message='{"requiredField":"test","optionalField":{"string": "teste"}}'

[sorend]

There is a ticket. Go vote:
https://issuetracker.google.com/issues/242757468

Due to this issue we are running pubsub schema-less until it is solved.