In case of ordering does PubSub honour any retry settings ?

[ankit-pradhan]

If ordering is enabled does PubSub honour any retry settings.
From below link we understand that regardless of retry settings it will continuously retry publishing the message.

https://cloud.google.com/pubsub/docs/publisher#retry_ordering

• 
  
  So while publishing, will it still consider any retry settings as below or it will disregard all the retry settings ?
  
  1 ) InitialRetryDelay
  2 ) RetryDelayMultiplier
  3 ) MaxRetryDelay
  4 ) InitialRpcTimeout
  5 ) RpcTimeoutMultiplier
  6 ) MaxRpcTimeout
  7 ) TotalTimeout
  8 ) MaxAttempts
  
  

[ms4446]

When ordering is enabled and a client library retries a request, the client library will repeatedly retry the request regardless of the retry settings. This means that the Pub/Sub system will not honor the specified retry settings such as:

1. InitialRetryDelay
2. RetryDelayMultiplier
3. MaxRetryDelay
4. InitialRpcTimeout
5. RpcTimeoutMultiplier
6. MaxRpcTimeout
7. TotalTimeout
8. MaxAttempts

The retry will continue indefinitely until the message is successfully published or a non-retryable error occurs​

[ankit-pradhan]

Hi @ms4446 ,

So I have few questions.
1) In com.google.cloud.pubsub.v1.Publisher.java we already have "Preconditions.checkArgument" method for some validations. 
setRetrySettings method is already using this method then why don't we put another check which can error out and say that if ordering is enabled you can't set retry settings ?
I think if we add this in next version it will help pubsub users.

"

[ms4446]

Here are some general considerations. 

---

@ankit-pradhan wrote:

Another question.

When I am using below settings it works well and we are able to publish messages with ordering enabled.

1. InitialRetryDelay = Duration.ofMillis(pubsubInitialRetryDelay)
2. RetryDelayMultiplier = 1.5
3. MaxRetryDelay = Duration.ofSeconds(pubsubMaxRetryDelay)
4. InitialRpcTimeout = Duration.ofSeconds(pubsubInitialRpcTimeout)
5. RpcTimeoutMultiplier = 1.5
6. MaxRpcTim

---

When designing APIs, it's important to balance user flexibility with enforcing constraints. In this case, allowing users to set retry settings even when ordering is enabled might have been a design choice made to give users more flexibility, as they might want to configure these settings for other use cases where ordering is not enabled.

As you suggested, adding a warning or exception when users attempt to set retry settings with ordering enabled could potentially save users from confusion. However, it could also break existing code if users are currently setting these values even when ordering is enabled. For such changes, developers need to carefully consider backwards compatibility and the potential impact on existing users.

[ankit-pradhan]

Hi @ms4446 

Another question.

When I am using below settings it works well and we are able to publish messages with ordering enabled.

1. InitialRetryDelay = Duration.ofMillis(pubsubInitialRetryDelay)
2. RetryDelayMultiplier = 1.5
3. MaxRetryDelay = Duration.ofSeconds(pubsubMaxRetryDelay)
4. InitialRpcTimeout = Duration.ofSeconds(pubsubInitialRpcTimeout)
5. RpcTimeoutMultiplier = 1.5
6. MaxRpcTimeout = Duration.ofSeconds(pubsubMaxRpcTimeout)
7. TotalTimeout = return Duration.ofSeconds(pubsubTotalTimeout)
8. MaxAttempts = 3

But If I change Duration.ofSeconds to Duration.ofMillis as below it starts failing repeatedly with grpc timeout errors and never able to publish messages.

1. InitialRetryDelay = Duration.ofMillis(pubsubInitialRetryDelay)
2. RetryDelayMultiplier = 1.5
3. MaxRetryDelay = Duration.ofMillis(pubsubMaxRetryDelay)
4. InitialRpcTimeout = Duration.ofMillis(pubsubInitialRpcTimeout)
5. RpcTimeoutMultiplier = 1.5
6. MaxRpcTimeout = Duration.ofMillis(pubsubMaxRpcTimeout)
7. TotalTimeout = return Duration.ofMillis(pubsubTotalTimeout)
8. MaxAttempts = 3

[ms4446]

The issue you're experiencing seems to be related to the change from seconds to milliseconds in your configuration. The gRPC timeout errors indicate that the client is not getting a response from the server within the timeframe specified by the RpcTimeout and TotalTimeout settings.

Remember that 1 second is equal to 1000 milliseconds. So, if you're changing Duration.ofSeconds to Duration.ofMillis without adjusting the actual values, you're effectively reducing the timeouts and delays by a factor of 1000. This could lead to timeout errors if the Pub/Sub service cannot complete operations within these shorter durations.

For example, if pubsubInitialRpcTimeout was 5 when you were using Duration.ofSeconds, this would translate to a timeout of 5 seconds. If you switch to Duration.ofMillis but keep pubsubInitialRpcTimeout as 5, you're now setting a timeout of just 5 milliseconds, which is likely too short for the Pub/Sub service to respond.

The gRPC timeouts are likely causing your publish calls to fail before they have a chance to succeed. To resolve this, you could adjust the values of the variables to account for the change in units. For instance, if you want a timeout of 5 seconds, you would set pubsubInitialRpcTimeout to 5000 when using Duration.ofMillis.

However, as previously mentioned, please note that these settings will not be honored when ordering is enabled. The client library will repeatedly retry the request, regardless of these settings​

[ankit-pradhan]

Hi @ms4446 ,

Good explanation. But with ordering being enabled it should disregard these retry settings and should behave same if someone configures it.
But it behaves differently in these two cases even when ordering is enabled in both the cases.
Looks like there are some footprints of these retry settings even when ordering is enabled.
What are your thoughts ?

[ankit-pradhan]

Hi ms4446,

It looks little contradicting in case of ordering enabled.
On 1 side we say that all these retry settings are disregarded in case of ordering enabled and on another side when we change ofSeconds to ofMillis it starts failing.

To me as per Publisher behaviour it is disregarding MaxAttempts retry setting and publishing continuously in case of retry-able errors.
But it is still considering other timeout retry settings.

What are your thoughts on this ?

[ankit-pradhan]

Hi @ms4446 ,

Google library allowing to setRetry settings even with ordering enabled at Publisher.java .
And after that some retry settings work and some fail with grpc timout errors creates a lot of unnecessary confusion.
I think in coming versions library should check it and fail fast with proper error message.

[ms4446]

I understand your frustration. The behavior you're describing can indeed be confusing.

While Google's Pub/Sub client libraries try to make common use cases straightforward, they must also handle a wide range of scenarios and use cases, which sometimes leads to complex behaviors.

[ankit-pradhan]

Hi @ms4446 ,

With all these questions and experience with ordering enabled it looks like , Yes Publisher disregards max attempts and retries continuously for retry-able errors but there are still some footprints on Publisher behaviour for other retry settings and behaves differently.
Correct me if I am missing something here.

[ms4446]

Pub/Sub client library continuously retries for retryable errors, regardless of the MaxAttempts setting. This makes sense, as the goal of enabling ordering is to ensure that all messages are published in the correct order, and stopping retries after a certain number of attempts could lead to messages being skipped, disrupting the order.

[ankit-pradhan]

Hi ms4446,

So can we conclude that MaxAttempts retry setting is disregarded but other timeout settings still have some foot prints and atleast update the google documentation ?

[ms4446 · Accepted solution]

Thanks for the feedback. We will raise with the product team to review the documentation to make more clear.