Payload formparam not assigning the correct payload

[darpanjain420]

I have got an assign message like below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage continueOnError="false" enabled="true" name="am-set-request">
<DisplayName>am-set-request</DisplayName>
<Properties/>
<Set>
<Headers>
<Header name="api-key">{apikeyVar}</Header>
<Header name="Content-Type">application/x-www-form-urlencoded</Header>
</Headers>
<QueryParams>
<QueryParam name="abd">{abdVar}</QueryParam>
<QueryParam name="abc">{abcVar}</QueryParam>
</QueryParams>
<FormParams>
<FormParam name="Payload">{request.content}</FormParam>
</FormParams>
</Set>
<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
<AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

but when I get a payload like below:

{
"ABC": {
"message": {
"origin": "",
},
"response": {
"messages": [
{
"messageId": "message123",
"message": "<ns0:ABC xmlns:ns0=\"urn:example.com:fin:12345:CUST\"><Tag0> <Tag1>12344</Tag1> <Tag2>B123456</Tag2> <Tag3/> <Tag4/> <Tag5>44444</Tag5> </Tag0></ns0:ABC>"
}
]
}
}
}

the apigee confuses the = sign and assumes there are 2 formparams. I had to use a workaround like below to resolve this:
context.setVariable("request.formparam.Payload",JSON.stringify(JSON.parse(context.getVariable("request.content"))));

Is there some better solution for this? or is the Payload formparam assignment in the AM policy wrong?
Thanks in advance.

[dchiesa1 · Accepted solution]

hmmm

what I understand you're doing:

1. you have an inbound request payload with content-type:application/json
2. you want to modify that request message so that the content-type becomes application/x-www-form-urlencoded
3. you want to take the entire original request request body content (request.content), which is json, and use that as the value for a single form parameter. Essentially replacing the json content with form content.
4. And you also want to add some query params
5. and then send that to some remote target

is that about right?

And also, the JSON actually wraps some XML!! Amazing! XML, wrapped in JSON, then encoded as a form param. (If there were an award for using the most data formats in one message, you would win!)

What you're doing is ... unorthodox. A little bit unique. First question: Are you sure you want to do it that way? If I were an architect on a project , in which someone proposed doing this, I would ... ask that person to re-consider. The issue you're running into is just ONE of the reasons why it's probably not the best idea. 

I tried what you tried, and it MOSTLY works. In fact I don't see a problem with the equals sign at all. You said "the apigee confuses the = sign and assumes there are 2 formparams." but that is a conclusion, not an observation. What did you observe that lead you to the conclusion that "apigee is confused"? My observation shows that the form param gets appropriately set. Subsequent to the AssignMessage, my request body looks like this:

Payload=%7B%20%20%22ABC%22%3A%20%7B%20%20%20%20%22message%22...

...which looks correct to me. All the stuff after the equals sign is an encoded JSON payload.

But there is one nuance. One thing I had to change. By using the Set element in AssignMessage, you are setting the content-type and setting the formparam. But you haven't removed the original json content. So there is still a JSON body on the message. You need to remove the original content, and then set the form param. You can remove the payload body within AssignMessage like this:

<AssignMessage continueOnError="false" enabled="true" name="AM-Transform-Message">
  ...
  <Remove>
    <Payload/>
  </Remove>

But removing the original content means it will no longer be available in the variable request.content. request.content will return the empty string. So you need to save the original content in a different variable, then remove it, then set the form param. like this:

<AssignMessage name="AM-Transform-Message">
  <AssignVariable>
    <Name>retained-payload</Name>
    <Ref>request.content</Ref>
  </AssignVariable>
  <Remove>
    <Payload/>
  </Remove>
  <Set>
    <Headers>
      <Header name="api-key">{apikeyVar}</Header>
      <Header name="Content-Type">application/x-www-form-urlencoded</Header>
    </Headers>
    <QueryParams>
      <QueryParam name="abd">{abdVar}</QueryParam>
      <QueryParam name="abc">{abcVar}</QueryParam>
    </QueryParams>
    <FormParams>
      <FormParam name="Payload">{retained-payload}</FormParam>
    </FormParams>
  </Set>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
  <!--
      OMIT THIS. IT DOES NOTHING.
  <AssignTo createNew="false" transport="http" type="request"/>
  -->
</AssignMessage>

[darpanjain420]

Please see the below screenshot of trace not showing "Payload=" in case of below request.content:
{
"ABC": {
"message": {
"origin": "",
},
"response": {
"messages": [
{
"messageId": "message123",
"message": "<ns0:ABC xmlns:ns0=\"urn:example.com:fin:12345:CUST\"><Tag0> <Tag1>12344</Tag1> <Tag2>B123456</Tag2> <Tag3/> <Tag4/> <Tag5>44444</Tag5> </Tag0></ns0:ABC>"
}
]
}
}
}

But in the case I remove the equal sign i get below trace:

where you can see the "Payload="

I tried removing the payload first and then assigning the data as per your code, I dont see the issue though.
I want to understand if this type of payload can or should be flagged in the JSON threat protection?
If yes then what tag do I use to enforce this?
@dchiesa1 

[dchiesa1]

OK I am not sure what problem we are solving here. 

You have pointed out that you see something in the trace UI that you don't expect. I think the thing that you see in the trace session is a side effect of not clearing the payload, before adding a Formparam.  But I am not sure; I haven't tested this myself in the old trace/debug experience. So here is what I suggest, If the problem you are concerned about is in the trace session, and you are seeing something you don't expect, please clarify that in your reply to this message. Also, test both scenarios - (1) assigning a formparam AFTER clearing the payload, and (2) assigning a formparam WITHOUT clearing the payload. Tell me what you observe for both of those cases, not in the trace session, but in the actual payload that gets assigned. You may need to rely on an echo service that tells you what was in the payload. Also explain if what you observe is surprising to you, and if so, in what way. 

In your reply, you are also raising what seems to be a very different question, regarding a JSON Threat Protection policy.  Is that the core question?    If so, maybe we should ignore the Trace UI/Debugsession issue.  If you're concerned about JSON Threat Protection, what policy configuration are you using, what results do you see, and what results  do you expect to see? 

Last thing - I feel that I responded to  your original question with some detail. Your response after that did not seem to acknowledge any of the information or suggestions I made. Are you reading what I wrote?  Can you confirm that you understand, that you tried what I suggested, and that the observations you then observe are consistent with what I predicted?  Alternatively, maybe you can confirm that you understand what I wrote, you tried what I suggested, and you saw results that do not align with what I predicted.  Maybe just let me know that we're on the right track, or not.  Respond to what I wrote. 

If you just respond by raising new questions which are unrelated, or just start providing new observations, then I'm not sure if what I'm writing here is landing, or making sense, or being understood.  In which case I will need to be clearer, I guess. 

thanks

[darpanjain420]

Hi
I think I did not clarify in my earlier response clearly but I tried the exact solution provided by you and it worked perfectly. Please note the following statement at the end of my reply which stated this:
I tried removing the payload first and then assigning the data as per your code, I dont see the issue though.

The further evidence I provided was to showcase the behaviour at my end which you mentioned you did not observe at your end. So I though of providing the evidence even though your workaround solution already worked for me.
I asked the JSON threat protection question to further secure my proxy so as to flag such requests in the future if they come.
Please note the following JSON threat protection policy is already implemented which did not catch this response:
<JSONThreatProtection continueOnError="false" enabled="true" name="jtp-request-threat-protection">
<DisplayName>jtp-request-threat-protection</DisplayName>
<Properties/>
<ObjectEntryNameLength>50</ObjectEntryNameLength>
<Source>request</Source>
</JSONThreatProtection>