Multiple appends to same record overwrite even when using transaction

[Ton1]

Hi all,

1

I am running Firestore in Datastore mode and have a structure in one of my tables where I use an array of items in each entity. I store the keys of these items in that array.

In one of my (cloud run) services I write a new entity (another kind) and add the key of these items to this array (on an existing entity). This write can happen from several places concurrently. I am using "allocate_ids" to pre-allocate the keys...

However sometimes one of my writes got over-written, even though I am using transactions; my code is as follows:

    # key = the key of the existing entity to which I want to add data
    # data = the data to put into the new item

    # We pre-allocate the keys (so we know what to store in the array)
    itemkeys = client.allocate_ids(client.key("item"), 1)
    itemkey = itemkeys[0]

    trans = client.transaction()
    trans.begin()

    # First handle item
    print(f"Writing to item key: {itemkey.id}")
    item = datastore.Entity(itemkey)
    item.update(data)

    # Then entity
    print(f"Adding to entity : {key.id}")
    entity = client.get(key)
    print(f"Current length: {len(entity['array'])}")
    entity["array"].append(itemkey)
    print(f"New length: {len(entity['array'])}")

    # Write the items
    trans.put(item)
    trans.put(entity)

    # Commit and pray
    trans.commit()

However, I still see occasions where the array misses elements. It looks like sometimes the element is taken from cache, even though I have put the "get" inside the transaction...

Can anyone provide some options/insights in what is going on? Am I trying somthing that is not supported?

N .b. I checked and the database is currently using "Pessimistic" concurrency. 

[marcanthonyb]

Hi @Ton1 ,

Have you enabled offline persistence mode in your Firestore? Taking data from the "cache" means that there are some moments where Firestore detects that it's offline. If possible, could you include a reproducible step on how the issue occurred and also, include some logs that you can get from Cloud Logging when the issue occurs.

Regards,
Marc

[Ton1]

Hi Marc,

Thanks for your response. I have a more detailed question posted on stackoverflow  (https://stackoverflow.com/questions/75790836/transactions-and-array-type-inconsistencies). I have lifted the code portion of that and have appended that (it's much more detailed) to the original question.

I am triggering this code from Google Cloud Run, so there is should be no mobile (possibly offline) component in this scenario.

The other problem is that it only occurs once in every ~50-70 writes, so I don't have a nice re-producable example. The code outlined is to 90% matching with my production-code, so it is a really good starting point...

Hope you can shed some more light on this issue...

Ton