Commas getting replaced by its encoded value %2C

[zaheen555]

Hello,

I have a requirement where I have to call a target endpoint with commas in its query string. The target url is set using 'target.url' in a js policy

For example. https:// example.com/v1/employee?prompt=100,200

This is getting replaced as https:// example.com/v1/employee?prompt=100%2C200

The backend server is not handling the encoded characters. Is there anyway this can be done in apigee?

I also tried decodeURI(targeturl) which also did not work.

Thanks

[dknezic · Accepted solution]

It's a reserved character and must be encoded so you will need to fix your backend.

https://datatracker.ietf.org/doc/html/rfc3986#section-2.2

[slatty]

Hi,

I am also having issues with this:

https://example.com/hostasset?fields=id,name

becomes

https://example.com/hostasset?fields=id%2Cname

According to the link you provided commas are used as delimiters and are in the same group as "=" and actually should NOT be encoded since it acts as a delimiter depending on the application.

"Percent- encoding a reserved character, or decoding a percent-encoded octet that corresponds to a reserved character, will change how the URI is interpreted by most applications. Thus, characters in the reserved set are protected from normalization and are therefore safe to be used by scheme-specific and producer-specific algorithms for delimiting data subcomponents within a URI."

[dchiesa1]

I guess the specifications are a bit loose, in order to allow different systems or schemes to treat different characters as delimiters. 

According to the spec, the value of every query parameter must be URI encoded.  (In JavaScript you can do this with the  builtin encodeURIComponent() function).  What we are discussing here is whether, during the encoding, a particular character, in this case ASCII 0x2C, is REQUIRED to be percent-encoded as part of that encoding.  But in any case, a sender (like Apigee) MAY percent-encode the character. More on that.

And whether or not Apigee encodes a comma as %2C, seems to me, to be mostly immaterial.  The spec states that each query param must be encoded; it follows then, that it is the responsibility of the receiving (server) side to DECODE each parameter. Whether or not one or another character within a parameter is encoded, should not matter so much.  Upon receiving "id%2Cname" as a query param value, the server should URI decode it to attain "id,name".  In a server implemented in JavaScript, you can do the decoding with decodeURIComponent(), but if you are using a node framework like express, express does the decoding automagically for you.  And at that point, after decoding, it is up to the server to decide what to do with the decoded value. 

So if your server is choking on the "id%2Cname" that Apigee is passing it, then I think the problem is with your server. It should decode properly. 

The only reasons I can think of, where this would be a problem is 1. if you have a very old server, for which you don't have the source code, in which case, I'm very sorry for your loss, but it's not a problem with Apigee.   Or 2. you are using Apigee to do something strange or impolite, in which case, please don't ask people here to help you with that. 

[slatty]

Hi Dino,

there is a third option which is my situation. My team and I are responsible for integrations with external applications so we have no control over the “backend” with this particular integration. This is the first we’ve encountered this issue and I was trying to achieve one of two things.

A. Figure out which application SHOULD be doing what. However, as you mentioned, the spec is loose so the responsibility could go either way.

B. Figure out if there is a workaround I can do in Apigee to accommodate this external application

I am attempting to contact The external application in question to see if this issue can be changed on their end as well but I’m not hopeful.

Thank you for your response.

[antfrasparkle]

Hello Dino,

you are right. The question is: how apigee managed MongoDB query, where in query string shuoud be a lot of characther that haven't to be decoded?

And so on, we have a SAAS product that explicit say

• The client must not encode special symbols used in Resource Query Language (RQL). For example, some REST clients replace parenthesis ‘(‘ and ‘)’ respectively for ‘%28’ and ‘%29’, which is not compatible with the RQL parser.

[dchiesa1]

we have a SAAS product that explicit say: The client must not encode special symbols used in Resource Query Language (RQL).

You have a SaaS product which says you must not encode parameters that explicitly MAY be encoded according to the HTTP Specs? Then I will go back to my earlier statement: I think the problem is with your server. It should decode properly.

how apigee managed MongoDB query, where in query string shuoud be a lot of characther that haven't to be decoded?

That seems like a conflict. You have two systems that observe opposite conventions. I don't have any suggestions for resolving that conflict.

[antfrasparkle]

We have multiple scenario in API integration

First scenario: SAAS platform that use RQL system. I can't say: hello xyz, please change your policy because we use APIGEEX

Second scenario: query on rest db (like but not only mongodb) with query like this

https://mydb-fafc.restdb.io/rest/people?q={"$and": [{"name": "Jane"}, {"last-name": "Cassidy"}]}

You are right when refer to HTTP Spec, but others APIGW solve this issue and surely APIGEE can't ignorate this kind of API rest calls

BR

[dchiesa1]

If you are using restdb.io, would the codehooks mechanism allow you to perform the required decoding on the restdb side? 

https://restdb.io/docs/codehooks

[antfrasparkle]

Sure, J can also trigger a cloud function to do something, but, in my opinion these are workaraound. It should be better APIGEEX add some features to solve these scenarious

Thx a lot

[dchiesa1]

Using the documented mechanism in restdb.io  to handle request decoding is "a workaround"?   I am  surprised by your reaction.

My view is that 98% of the internet encodes commas as %2C, and here you have called my attention to a system, restdb.io ,  that assumes commas are not so encoded.  But that system has a mechanism to handle this, and you can easily adjust that system with a tiny hook, to decode commas . And that would fix it.  Why would that be unacceptable?  Why should you want Apigee to solve the problem, specifically for restdb, when restdb has the mechanism to solve the problem, and it's documented and available? 

(You don't need to answer that question. It just feels very curious to me.)