I've got an issue with encoding. The issue is that when an email with + is added as queryparam in AssignMessage, Apigee encodes + as 20 rather than 2B. I suppose the issue will be any queryparam with + in it. Is this correct behavior? What is the fix/workaround?
get encoded as john.doe%20test%40gmail.com when calling backend rather than john.doe%2Btest%40gmail.com
<Set> <QueryParams> <QueryParam name="id">john.doe+test@gmail.com</QueryParam> </QueryParams> </Set>
We are using Apigee edge cloud
Hmmmmmm
I don't see the behavior you're describing.
Here's the result of a proxy I'm using:
$ curl -i https://$ORG-$ENV.apigee.net/assignmsg-2/f1 HTTP/1.1 200 OK Date: Mon, 18 Nov 2019 16:30:53 GMT Content-Type: application/json; charset=utf-8 Content-Length: 932 Connection: keep-alive Vary: Accept-Encoding ETag: W/"3a4-yqrLqX9QMRgKl8Y4wfQ1NkL/T/4" X-Cloud-Trace-Context: 19291c091702b1c32e93af4afdf9d086 Server: Google Frontend { "url": "/?id=john.doe%2Btest%40gmail.com", "method": "GET", "headers": { "host": "buoyant-climate-208500.appspot.com", "x-forwarded-for": "104.132.51.85,34.82.153.38, 169.254.1.1", "x-forwarded-proto": "https", .... }, "body": {} }
Which looks to me like... the backend got john.doe%2Btest%40gmail.com
..which is what you wanted. Right?
Here's the policy I used in the request flow.
<AssignMessage name='AM-QueryParam-Unencoded'> <Set> <QueryParams> <QueryParam name="id">john.doe+test@gmail.com</QueryParam> </QueryParams> </Set> </AssignMessage>
Thanks for your response. It is strange. Attached is the proxy I used. Do you see anything unusual?
In trace it shows following after AssignMessage step
I invoked the end-point using postman and had header Content-Type = application/json; chartset=utf-8
If invoked with header header (without utf-8) Content-Type = application/json then encoding is okecho2-rev1-2019-11-18.zip
There was similar issue in past https://community.apigee.com/questions/38149/apigee-url-encodes-plus-sign-incorrectly.html
ack
I can't download your attachment. I get an "access forbidden" message.
Can you try attaching again? In the meantime, here's mine.
echo2-rev1-2019-11-18.zip
if you cann't download, then will copy paste policy etc. Its pretty small
My proxy is very basic. Checked yours and saw use of "encodeHTML" . Tried that and it does encoding of encoded.
FYI - we are using cloud based Apigee
Try both flows. The f1 flow does not use encodeHTML. If I have understood what you want, that example works for you. The backend receives
/?id=john.doe%2Btest%40gmail.com
...which is what you wanted. Right?
With
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <AssignMessage async="false" continueOnError="false" enabled="true" name="Assign-Message-1"> <DisplayName>Assign Message-1</DisplayName> <Properties/> <Set> <QueryParams> <QueryParam name="id">{encodeHTML('john.doe+test@gmail.com')}</QueryParam> </QueryParams> <FormParams/> </Set> <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables> <AssignTo createNew="false" transport="http" type="request"/> </AssignMessage>
I get (seems like encoding of encoded)
Assign Message-1 GET /echo2?id=john.doe%26%23x2b%3Btest%26%23x40%3Bgmail.com
with
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <AssignMessage async="false" continueOnError="false" enabled="true" name="Assign-Message-1"> <DisplayName>Assign Message-1</DisplayName> <Properties/> <Set> <QueryParams> <QueryParam name="id">john.doe+test@gmail.com</QueryParam> </QueryParams> <FormParams/> </Set> <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables> <AssignTo createNew="false" transport="http" type="request"/> </AssignMessage>
I get
Assign Message-1 GET /echo2?id=john.doe%20test%40gmail.com
"I get" means... what?
Where do you "get", or see, this information?
Are you observing this in the Trace UI? Or are you seeing this in the upstream/backend ?
If you are observing this in the trace UI, don't rely on it. Check what the upstream actually receives.
2nd. Have you tried my proxy?
I've told you it works and I uploaded it. Did you try it? What results did you see? Do you see the results that I see?
results.txtThanks for response.
1. I did upload the proxy you provided and it is not working as expected. Please try it out with URLs in the file. Seems that Content-Type changes how encoding of url is done. Is that correct?
2. My observations that I put it under "I get" were from Trace UI
3 URLs and results in attached file. Please try it out and see what results you observe
What problem are we solving? I've lost track.
I thought your goal was to accomplish a specific thing, specifically: find the answer to the question, "how can I send as a query parameter, an email that has an embedded plus sign?" I think I answered that question. Have we solved the problem?
If that is not what you're trying to do, then what?
I see you have conducted a bunch of different tests. What is your goal here?
The objective still the same - being able to have query param (email) with plus sign in value. The problem is/was that when request is made with header Content-Type: application/json; charset=utf-8, the encoding for plus sign is incorrect.
Problem: query param with value john.doe+test@gmail.com
The problem is not solved. I have tried your proxy (please see attached file with result.txt).
The purpose of tests was to illustrate the issue.
I think you're saying,
if you send a GET request with a Content-type header, then ... Apigee does not behave the way it ought to behave.
That sounds like a weird bug in Apigee. I've created a bug on your behalf. b/145164863
A suggested workaround to that is:
Don't send a content-type header with a request that has no content.
Content-Type is intended to communicate the type of the payload, and as you know there is no payload in a GET, therefore the content-type header is not needed, and some might say it is incorrect for a client to include it in a GET request.
But it's possible the behavior you are describing might also apply to POST requests. And in any case it is also true that the encoding of query params sent from Apigee to a target should not be dependent upon the Content-type header sent inbound.
Hi @dino-at-Google,
Is this issue fixed in Apigee?
This can be fixed in MessageProcessor level, can we fix it in proxy level?
Concern - Query param contains the "%" char in the value but Apigee is encoding it to "%25".
%25 is the correct hex for a % character.
@dane knezic - My ask is Apigee should pass "%" as is to the target endpoint without converting it to "%25". Hope you understand my concern?
That would be the wrong thing to do, please refer to https://tools.ietf.org/html/rfc3986#section-2.1
I agree with Dane. Also, if you have a new question you should ask a new question. If you like you can reference this existing one, if it's relevant and similar.
Otherwise your question and the back-and-forth gets buried in a deep thread of comments. That's not helpful for anyone.
If you have a new question, please ask a new question.
Kudos to you for curating the old stuff!
Salute to the maintainers!
Is there any fix for this. I am still getting this issue. With Content-Type: application/json; charset=utf-8, '+' sign is getting encoded to %20 instead of %2B even in POST method as well.
Hello @deepchandra-au!
Thanks for reaching out to the community, I want to make sure your question gets the attention it deserves.
I highly recommend you open a new thread specifically for your situation. When you do, please reference this current thread so the community has all the context. This will help your question get appropriate visibility and allow other members to chime in with guidance.
We appreciate you reaching out!
I've revisited this and I am still seeing what is perceived to be improperly encoded queryparams, even after finding the docs that are supposed to control this behavior in ProxyEndpoint configuration elements.
I tried changing the property and various tests with no ability to correct the behavior.
<HTTPProxyConnection>
<BasePath>/notarget</BasePath>
<Properties>
<Property name="request.queryparams.ignore.content.type.charset">true</Property>
</Properties>
</HTTPProxyConnection>
I am always seeing the "request.queryparam.email" variable with a value of "kurt any@foo.com" when I send "?email=kurt+any@foo.com" no matter what I send in content-type header.
After further digging, the root cause is the underlying Java URLDecoder.decode function. Given the following test code:
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.io.UnsupportedEncodingException; // Import the exception class
import java.nio.charset.StandardCharsets;
public class UrlEncodeDecodeExample {
public static void main(String[] args) {
String encodedValue = "name+suffix@any.com";
try {
System.out.println(); // Add a blank line for better readability
String defaultEncodedValue = URLEncoder.encode(encodedValue, StandardCharsets.UTF_8.toString());
System.out.println("Encoded incorrectly encoded value: " + encodedValue + " becomes: " + defaultEncodedValue);
System.out.println(); // Add a blank line for better readability
// Decoding with UTF-8, which is generally recommended
String decodedValue = URLDecoder.decode(encodedValue, StandardCharsets.UTF_8.toString());
System.out.println("Incorrectly encoded value: " + encodedValue);
System.out.println("Decoded incorrectly encoded value: " + decodedValue);
} catch (UnsupportedEncodingException e) {
// This catch block handles the exception if the charset is not supported
System.err.println("Error: The specified encoding is not supported.");
e.printStackTrace(); // Print the stack trace for debugging
}
System.out.println(); // Add a blank line for better readability
String correctlyEncodedValue = "name%2Bsuffix%40any.com"; // %2B for +, %40 for @
try {
System.out.println(); // Add a blank line for better readability
String correctlyDecodedValue = URLDecoder.decode(correctlyEncodedValue, StandardCharsets.UTF_8.toString());
System.out.println("Correctly encoded value: " + correctlyEncodedValue);
System.out.println("Decoded correctly decoded value: " + correctlyDecodedValue);
} catch (UnsupportedEncodingException e) {
// This catch block handles the exception for the second decode call
System.err.println("Error: The specified encoding is not supported.");
e.printStackTrace();
}
}
}
Running that test case produces the output:
Encoded incorrectly encoded value: name+suffix@any.com becomes: name%2Bsuffix%40any.com
Incorrectly encoded value: name+suffix@any.com
Decoded incorrectly encoded value: name suffix@any.com
Correctly encoded value: name%2Bsuffix%40any.com
Decoded correctly decoded value: name+suffix@any.com
Bottom line, to use a literal '+' character in a query param value percent encode it as '%2B'.