Announcements
This site is in read only until July 22 as we migrate to a new platform; refer to this community post for more details.
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Apigee URL encoding issue

I've got an issue with encoding. The issue is that when an email with + is added as queryparam in AssignMessage, Apigee encodes + as 20 rather than 2B. I suppose the issue will be any queryparam with + in it. Is this correct behavior? What is the fix/workaround?

john.doe+test@gmail.com

get encoded as john.doe%20test%40gmail.com when calling backend rather than john.doe%2Btest%40gmail.com

<Set>
  <QueryParams>
    <QueryParam name="id">john.doe+test@gmail.com</QueryParam>
  </QueryParams>
</Set>

We are using Apigee edge cloud

0 23 4,349
23 REPLIES 23

Hmmmmmm

I don't see the behavior you're describing.

Here's the result of a proxy I'm using:

$ curl -i https://$ORG-$ENV.apigee.net/assignmsg-2/f1
HTTP/1.1 200 OK
Date: Mon, 18 Nov 2019 16:30:53 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 932
Connection: keep-alive
Vary: Accept-Encoding
ETag: W/"3a4-yqrLqX9QMRgKl8Y4wfQ1NkL/T/4"
X-Cloud-Trace-Context: 19291c091702b1c32e93af4afdf9d086
Server: Google Frontend
{
  "url": "/?id=john.doe%2Btest%40gmail.com",
  "method": "GET",
  "headers": {
    "host": "buoyant-climate-208500.appspot.com",
    "x-forwarded-for": "104.132.51.85,34.82.153.38, 169.254.1.1",
    "x-forwarded-proto": "https",
     ....
  },
  "body": {}
}

Which looks to me like... the backend got john.doe%2Btest%40gmail.com

..which is what you wanted. Right?

Here's the policy I used in the request flow.

<AssignMessage name='AM-QueryParam-Unencoded'>
  <Set>
    <QueryParams>
      <QueryParam name="id">john.doe+test@gmail.com</QueryParam>
    </QueryParams>
  </Set>
</AssignMessage>

Thanks for your response. It is strange. Attached is the proxy I used. Do you see anything unusual?

In trace it shows following after AssignMessage step

GET /echo2?id=john.doe%20test%40gmail.com

I invoked the end-point using postman and had header Content-Type = application/json; chartset=utf-8

If invoked with header header (without utf-8) Content-Type = application/json then encoding is okecho2-rev1-2019-11-18.zip

GET /echo2?id=john.doe%2Btest%40gmail.com

There was similar issue in past https://community.apigee.com/questions/38149/apigee-url-encodes-plus-sign-incorrectly.html

ack

I can't download your attachment. I get an "access forbidden" message.

Can you try attaching again? In the meantime, here's mine.

apiproxy-assignmessage-encoding.zip

echo2-rev1-2019-11-18.zip

if you cann't download, then will copy paste policy etc. Its pretty small

My proxy is very basic. Checked yours and saw use of "encodeHTML" . Tried that and it does encoding of encoded.

FYI - we are using cloud based Apigee

Try both flows. The f1 flow does not use encodeHTML. If I have understood what you want, that example works for you. The backend receives

/?id=john.doe%2Btest%40gmail.com

...which is what you wanted. Right?

With

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage async="false" continueOnError="false" enabled="true" name="Assign-Message-1">
    <DisplayName>Assign Message-1</DisplayName>
    <Properties/>
    <Set>
        <QueryParams>
            <QueryParam name="id">{encodeHTML('john.doe+test@gmail.com')}</QueryParam>
        </QueryParams>
        <FormParams/>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

I get (seems like encoding of encoded)

Assign Message-1
GET /echo2?id=john.doe%26%23x2b%3Btest%26%23x40%3Bgmail.com

with

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage async="false" continueOnError="false" enabled="true" name="Assign-Message-1">
    <DisplayName>Assign Message-1</DisplayName>
    <Properties/>
    <Set>
        <QueryParams>
            <QueryParam name="id">john.doe+test@gmail.com</QueryParam>
        </QueryParams>
        <FormParams/>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
    <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

I get


Assign Message-1
GET /echo2?id=john.doe%20test%40gmail.com



"I get" means... what?

Where do you "get", or see, this information?

Are you observing this in the Trace UI? Or are you seeing this in the upstream/backend ?

If you are observing this in the trace UI, don't rely on it. Check what the upstream actually receives.

2nd. Have you tried my proxy?

I've told you it works and I uploaded it. Did you try it? What results did you see? Do you see the results that I see?

results.txtThanks for response.

1. I did upload the proxy you provided and it is not working as expected. Please try it out with URLs in the file. Seems that Content-Type changes how encoding of url is done. Is that correct?

2. My observations that I put it under "I get" were from Trace UI

3 URLs and results in attached file. Please try it out and see what results you observe

results.txt

What problem are we solving? I've lost track.

I thought your goal was to accomplish a specific thing, specifically: find the answer to the question, "how can I send as a query parameter, an email that has an embedded plus sign?" I think I answered that question. Have we solved the problem?

If that is not what you're trying to do, then what?

  • is your goal to run through a set of test cases and check whether the results fit your expectations?
  • Something else?

I see you have conducted a bunch of different tests. What is your goal here?

The objective still the same - being able to have query param (email) with plus sign in value. The problem is/was that when request is made with header Content-Type: application/json; charset=utf-8, the encoding for plus sign is incorrect.

Problem: query param with value john.doe+test@gmail.com

john.doe%20test%40gmail.com (this is what we get -incorrect value)

john.doe%2Btest%40gmail.com (it should be this value - correct value)

The problem is not solved. I have tried your proxy (please see attached file with result.txt).

results.txt

The purpose of tests was to illustrate the issue.

I think you're saying,

if you send a GET request with a Content-type header, then ... Apigee does not behave the way it ought to behave.

That sounds like a weird bug in Apigee. I've created a bug on your behalf. b/145164863

A suggested workaround to that is:

Don't send a content-type header with a request that has no content.

Content-Type is intended to communicate the type of the payload, and as you know there is no payload in a GET, therefore the content-type header is not needed, and some might say it is incorrect for a client to include it in a GET request.

But it's possible the behavior you are describing might also apply to POST requests. And in any case it is also true that the encoding of query params sent from Apigee to a target should not be dependent upon the Content-type header sent inbound.

Hi @dino-at-Google,


Is this issue fixed in Apigee?

This can be fixed in MessageProcessor level, can we fix it in proxy level?


Concern - Query param contains the "%" char in the value but Apigee is encoding it to "%25".


%25 is the correct hex for a % character.

@dane knezic - My ask is Apigee should pass "%" as is to the target endpoint without converting it to "%25". Hope you understand my concern?

That would be the wrong thing to do, please refer to https://tools.ietf.org/html/rfc3986#section-2.1

I agree with Dane. Also, if you have a new question you should ask a new question. If you like you can reference this existing one, if it's relevant and similar.

Otherwise your question and the back-and-forth gets buried in a deep thread of comments. That's not helpful for anyone.

If you have a new question, please ask a new question.

 

I know this is old, but, someone else asked about Apigee X always encoding the query params sent to a backend service.
 
When you create a query param with a special character, Apigee URL encodes it before sending it to the backend.
After trying a few things, it appears that Apigee cannot be coerced into behaving like an "improper" client.
Using an Assign Message correctly encodes the query param: "id=jane+doe@any.com" to: "id=john.doe%2Btest%40gmail.com"
 
I even tried setting the target.url directly, which correctly sets the value in DEBUG to:
"https://mocktarget.apigee.net/echo?id=jane+doe@any.com"
But the URL actually sent becomes encoded and low and behold, incorrectly to:
"https://mocktarget.apigee.net/echo?id=jane%20doe%40any.com"
Notice it is using "%20" instead of "%2B" 
 
Bottom line, use Assign Message to set query params and expect that Apigee will properly URL encode them, as any "proper" client should.

Kudos to you for curating the old stuff!  

Salute to the maintainers! 

Is there any fix for this. I am still getting this issue. With Content-Type: application/json; charset=utf-8, '+' sign is getting encoded to %20 instead of %2B even in POST method as well.

Hello @deepchandra-au!

Thanks for reaching out to the community, I want to make sure your question gets the attention it deserves.

I highly recommend you open a new thread specifically for your situation. When you do, please reference this current thread so the community has all the context. This will help your question get appropriate visibility and allow other members to chime in with guidance.

We appreciate you reaching out! 

I've revisited this and I am still seeing what is perceived to be improperly encoded queryparams, even after finding the docs that are supposed to control this behavior in ProxyEndpoint configuration elements.

I tried changing the property and various tests with no ability to correct the behavior.

 

  <HTTPProxyConnection>
    <BasePath>/notarget</BasePath>
    <Properties>
      <Property name="request.queryparams.ignore.content.type.charset">true</Property>
    </Properties>
  </HTTPProxyConnection>

 

I am always seeing the "request.queryparam.email" variable with a value of "kurt any@foo.com" when I send "?email=kurt+any@foo.com" no matter what I send in content-type header.

After further digging, the root cause is the underlying Java URLDecoder.decode function. Given the following test code:

import java.net.URLDecoder;
import java.net.URLEncoder;
import java.io.UnsupportedEncodingException; // Import the exception class
import java.nio.charset.StandardCharsets;

public class UrlEncodeDecodeExample {
    public static void main(String[] args) {
        String encodedValue = "name+suffix@any.com";

        try {
            System.out.println(); // Add a blank line for better readability

            String defaultEncodedValue = URLEncoder.encode(encodedValue, StandardCharsets.UTF_8.toString());
            System.out.println("Encoded incorrectly encoded value: " + encodedValue + " becomes: " + defaultEncodedValue);

            System.out.println(); // Add a blank line for better readability

            // Decoding with UTF-8, which is generally recommended
            String decodedValue = URLDecoder.decode(encodedValue, StandardCharsets.UTF_8.toString());
            System.out.println("Incorrectly encoded value: " + encodedValue);
            System.out.println("Decoded incorrectly encoded value: " + decodedValue);
        } catch (UnsupportedEncodingException e) {
            // This catch block handles the exception if the charset is not supported
            System.err.println("Error: The specified encoding is not supported.");
            e.printStackTrace(); // Print the stack trace for debugging
        }

        System.out.println(); // Add a blank line for better readability

        String correctlyEncodedValue = "name%2Bsuffix%40any.com"; // %2B for +, %40 for @
        try {
            System.out.println(); // Add a blank line for better readability
            
            String correctlyDecodedValue = URLDecoder.decode(correctlyEncodedValue, StandardCharsets.UTF_8.toString());
            System.out.println("Correctly encoded value: " + correctlyEncodedValue);
            System.out.println("Decoded correctly decoded value: " + correctlyDecodedValue);
        } catch (UnsupportedEncodingException e) {
            // This catch block handles the exception for the second decode call
            System.err.println("Error: The specified encoding is not supported.");
            e.printStackTrace();
        }
    }
}

Running that test case produces the output:

Encoded incorrectly encoded value: name+suffix@any.com becomes: name%2Bsuffix%40any.com

Incorrectly encoded value: name+suffix@any.com
Decoded incorrectly encoded value: name suffix@any.com

Correctly encoded value: name%2Bsuffix%40any.com
Decoded correctly decoded value: name+suffix@any.com

Bottom line, to use a literal '+' character in a query param value percent encode it as '%2B'.

References: RFC 3986 and RFC 1866.