ExtractVariables Policy Returns Array Even with index="0" — Nested JSONPath Filtering Issue

[runawaybon]

Issue Summary:
I'm using the ExtractVariables policy in Apigee X to extract values from a nested JSON structure using JSONPath with filters. Even when I use index="0" in the <Variable> tag, Apigee still returns the result as an array (e.g., ["Value1"]) instead of a single string ("Value1").

Sample JSON:

{
  "fieldA": "DateTime",
  "fieldB": "DateTime",
  "productType": [
    {
      "fieldC": "DateTime",
      "fieldD": "DateTime",
      "extensions": [
        { "key": "Key1", "value": "Value1" },
        { "key": "Key2", "value": "Value2" },
        { "key": "Key3", "value": "Value3" }
      ],
      "productInstances": [
        { "code": "code1", "url": "https://url/1" },
        { "code": "code2", "url": "https://url/2" }
      ]
    }
  ]
}

I want to extract the value corresponding to "key": "Key1" from the extensions array using the following policy:

<ExtractVariables name="Extract-Key1">
    <Source>response.content</Source>
    <JSONPayload>
        <Variable name="key1Value" index="0">
            <JSONPath>$.productType[*].extensions[?(@.key == 'Key1')].value</JSONPath>
        </Variable>
    </JSONPayload>
</ExtractVariables>

What I Get:

Even with index="0", the variable key1Value is set to:

["Value1"]

I expect it to be just "Value1".

Am I missing something or is this the expected behavior in Apigee X?

Any insights on how to achieve this would be greatly appreciated!

CC: @amitkhosla 

[paul-wright · Accepted solution]

Hi,

The array formatting is a result of using a Filter with jsonpath. This is the standard behavior of jsonpath when using a filter, use of a filter always results in an array. However, there is an optional parameter that can be set in for the formal jsonpath function which is array=true|false. 

To access the optional parameter in Apigee, the full jsonpath function must be used in a message template structure... see: 

https://cloud.google.com/apigee/docs/api-platform/reference/message-template-intro#json-path-functio...

Therefore an example that pulls the needed value, "Value1" from the provided example payload could look like the following using the AssignMessage -> SetVariable approach (note: I'm using the Request payload)

Note the false parameter at the end of jsonpath(path, flow variable, array?)

<AssignMessage continueOnError="false" enabled="true" name="AM-1">
<DisplayName>AM-1</DisplayName>
<Properties/>
<AssignVariable>
<Name>my-extracted-value</Name>
<Template>{jsonPath($.productType[*].extensions[?(@.key == 'Key1')].value,request.content,false)}</Template>
</AssignVariable>
<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
<AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

[paul-wright · Accepted solution]

Hi,

The array formatting is a result of using a Filter with jsonpath. This is the standard behavior of jsonpath when using a filter, use of a filter always results in an array. However, there is an optional parameter that can be set in for the formal jsonpath function which is array=true|false. 

To access the optional parameter in Apigee, the full jsonpath function must be used in a message template structure... see: 

https://cloud.google.com/apigee/docs/api-platform/reference/message-template-intro#json-path-functio...

Therefore an example that pulls the needed value, "Value1" from the provided example payload could look like the following using the AssignMessage -> SetVariable approach (note: I'm using the Request payload)

Note the false parameter at the end of jsonpath(path, flow variable, array?)

<AssignMessage continueOnError="false" enabled="true" name="AM-1">
<DisplayName>AM-1</DisplayName>
<Properties/>
<AssignVariable>
<Name>my-extracted-value</Name>
<Template>{jsonPath($.productType[*].extensions[?(@.key == 'Key1')].value,request.content,false)}</Template>
</AssignVariable>
<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
<AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

[dchiesa1]

I don't see any index attribute documented for that element. Where did you get that? I don't think that will do anything.

With the payload and the query expression you presented, the JSONPath query is expected to be ["Value1"]

JSONPath doesn't support indexing into an array result. In other words, you cannot use this:

$.productType[*].extensions[?(@.key == 'Key1')].value[0]

 You can view issue 806 on this, for jsonPath.

The way to solve this is one of these:

• use two ExtractVariables policies
• Use one AssignMessage, with multiple jsonPath expressions

[paul-wright]

Unfortunately,

<JSONPath>$.productType[*].extensions[?(@.key == 'Key1')].value[0]</JSONPath>

returns:  [ ]

Agree, the other way to solve this is with using a chain of calls as you have suggested.

[runawaybon]

Thanks a lot @paul-wright @dchiesa1 for describing and presenting with multiple alternatives to achieve the said solution.

---

@dchiesa1 wrote:

JSONPath doesn't support indexing into an array result. In other words, you cannot use this:

---

I agree. I had tried this out initially but as @paul-wright mentioned, it returned an empty array.

Seems like opting for multiple jsonPath expressions within an AssignMessage policy is the way to go in this scenario.

I tried the following policy snippet and this resolves the use case in hand.

<Template>{jsonPath($.productType[*].extensions[?(@.key == 'Key1')].value,request.content,false)}</Template>

Just one last thing: Is there a way I can dynamically set the 'key' I wish to check for within my jsonPath template?

Like in the snippet mentioned above, can I write: [?(@.key == '{key}')] where the {key} is actually a context variable available at API proxy runtime?

[paul-wright · Accepted solution]

Yes, there is a trick that often works within Apigee. This works really well with regex expressions and it works for the jsonpath. What you do is build the path in another variable. Then use the variable that has the constructed path in it. 

In this example, I pull 2 values from the example json. The original value for Key1 is still being pulled. I have added a second extraction that pulls the value for Key2 using the name "Key2" to dynamically build the json path.

<AssignMessage continueOnError="false" enabled="true" name="AM-1">
<DisplayName>AM-1</DisplayName>
<Properties/>

<!-- this could be provided by an existing variable... just setting it here for a test -->
<AssignVariable>
  <Name>value-to-search-for</Name>
  <Value>Key2</Value>
</AssignVariable>

<!-- Dynamically build the json path to use -->
<AssignVariable>
  <Name>json-path-to-use</Name>
  <Template>$.productType[*].extensions[?(@.key == '{value-to-search-for}')].value</Template>
</AssignVariable>

<AssignVariable>
  <Name>dynamically-extracted-value</Name>
  <Template>{jsonPath(json-path-to-use,request.content,false)}</Template>
</AssignVariable>

<!-- this is the previous lookup -->
<AssignVariable>
  <Name>my-extracted-value</Name>
  <Template>{jsonPath($.productType[*].extensions[?(@.key == 'Key1')].value,request.content,false)}</Template>
</AssignVariable>

<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
<AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

.

.

[dchiesa1]

YES

And one more note. There's a bug (internal ref b/416576251, pending fix): The Apigee Message Template does not correctly resolve constants like "true" and "false" when those terms are used as arguments to functions. So, an expression like this:

<AssignVariable>
  <Name>dynamically-extracted-value</Name>
  <Template>{jsonPath(json-path-to-use,request.content,false)}</Template>
</AssignVariable>

...will actually NOT resolve that last parameter to false, unless you also have this:

<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>

That's ok if you want the argument to resolve to false, but sometimes you want true. Also, setting that "IgnoreUnresolvedVariables" will apply to every variable reference, which is probably not what you want. To avoid this, for now, you can set a variable to the value of false or true. Like this:

<IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables> <!-- the default -->

<AssignVariable>
  <Name>want-array</Name>
  <Value>false</Value>
</AssignVariable>

<AssignVariable>
  <Name>dynamically-extracted-value</Name>
  <Template>{jsonPath(json-path-to-use,request.content,want-array)}</Template>
</AssignVariable>

And soon, the fix for this will be available in X and hybrid. At that point, the Apigee Message Template will correctly resolve constants like "true" and "false" when those terms are used as arguments to functions.

[runawaybon]

 Thanks again for the prompt response @paul-wright @dchiesa1 

Using the method as suggested, I was able to read the variable(s) dynamically within the same policy and build the jsonPath as required.