LookupCache retrieves inconsistent values on multiple requests.

[harrycho]

I have a fairly simple PopulateCache policy in one of the proxy as follows. It simply caches the current timestamp for a given user specified in KeyFragment.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<PopulateCache async="false" continueOnError="false" enabled="true" name="blacklist-user-PC">
    <DisplayName>blacklist-user-PC</DisplayName>
    <Properties/>
    <CacheKey>
        <KeyFragment ref="blacklist.user_id"/>
    </CacheKey>
    <CacheResource>user_blacklist</CacheResource>
    <Scope>Global</Scope>
    <ExpirySettings>
        <TimeoutInSec>86400</TimeoutInSec>
    </ExpirySettings>
    <Source>blacklist.timestampInSec</Source>
</PopulateCache>

Then in another proxy, I have a LookupCache policy that retrieves the cached value where the KeyFragment is also the user.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LookupCache async="false" continueOnError="false" enabled="true" name="lookup-cache-debug-LC">
    <DisplayName>lookup-cache-debug-LC</DisplayName>
    <Properties/>
    <CacheKey>
        <KeyFragment ref="request.formparam.user_id"/>
    </CacheKey>
    <CacheResource>user_blacklist</CacheResource>
    <Scope>Global</Scope>
    <AssignTo>flowVar</AssignTo>
</LookupCache>

When I hit PopulateCache proxy 3 times with the same key fragment, I get an ascending timestamp values and last one should get stored replacing the older values.

However, when I call the LookupCache proxy and call it multiple times, I get differing values rather than a last stored value.

I initially thought this was because of availability over consistency in CAP theorem but it never seem to eventually consistent.

I confirmed that LookupCache always uses the same "cachekey" and "cachename" in the trace UI.

For example:

store A 1 
store A 2
store A 3

// (order may differ)

get A -> 2 (expecting 3)
get A -> 1 (expecting 3)
get A -> 3


invalidate A

// (order may differ)

get A -> 1 (expecting cache miss)
get A -> (cache miss)
get A -> 2 (expecting cache miss)

Any idea what's going on here?

Thanks