Datastore concurrency mode migration impact

[manansaraf16]

Hey,
We have decided to migrate our datastore's concurrency mode from OPTIMISTIC_WITH_ENTITY_GROUPS to OPTIMISTIC concurrency mode. The main reason for that being that we want to enable TTL on our platform, and our not very dependent on entity group relationships in our databases.

I wanted to ask if anybody has faced any unknown 500 errors/bugs/database transaction failures while migrating their datastore concurrency mode on a production environment where they are receiving a lot of user traffic?
Thank you 

[Joy_S · Accepted solution]

Hi @manansaraf16,

Welcome to Google Cloud Community!

Migrating your datastore’s concurrency mode in a high-traffic production environment can present some challenges, so it’s great that you're thinking ahead about potential issues. Here are some of the challenges you may encounter during migration

• Increased Contention: Switching to OPTIMISTIC concurrency can lead to more conflicts if multiple transactions are trying to modify the same entity. This could cause retries, especially if your datastore relies on distributed transactions.
• Transaction Failures: If a transaction fails, it will often retry, which can lead to 500 errors if retries aren’t handled properly. Frequent conflicts in a busy environment can cause these retries to pile up, leading to bigger issues.
• Locking and Timeouts: Even though OPTIMISTIC mode reduces locking, heavy write operations could still cause timeouts or contention issues, especially if retries aren’t managed well.
• TTL Issues: If you’re enabling TTL, the main risk is making sure it works smoothly during updates or deletions. Migration mistakes could cause entities to expire too soon or stay too long.
• Performance: The change might impact performance, especially if you have many writes happening at once. Increased conflict resolution could lead to delays and retries.

Switching to OPTIMISTIC concurrency can help with enabling TTL, but it might bring up challenges like conflicts, retries, and 500 errors. You may also refer to this post for more information regarding concurrency modes. With careful testing, error handling, and monitoring, you can minimize risks and ensure a smooth transition. 

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.