Request header Content-Encoding with value "UTF-8" creates an unexpected error in Apigee

[shrenikkumar-s]

Hi All (again :))

Whenever a request with header Content-Encoding is sent Apigee just fails to parse the request and doesn't execute the PreFlow and throws an error abruptly stopping the flow:

{"fault":{"faultstring":"Unsupported Encoding \"UTF-8\"","detail":{"errorcode":"protocol.http.UnsupportedEncoding"}}}

Is this supposed/expected to be like this?

We did fair bit of reading & exploring the community as well and understand that: The issue seems to be is the header: Content-Encoding: UTF-8

As per RFC specification ( https://tools.ietf.org/html/rfc2616#section-3.5 ) value UFT-8 should not be present in Content-Encoding header as it is not valid and message-processor behaviour is correct.

Source/references:

https://www.googlecloudcommunity.com/gc/Apigee/protocol-http-UnsupportedEncoding/td-p/70917
https://www.googlecloudcommunity.com/gc/Apigee/How-to-fix-Unsupported-Encoding-quot-UTF-8-quot-error...
https://stackoverflow.com/questions/17154967/is-content-encoding-being-set-to-utf-8-invalid/17155003...
https://en.wikipedia.org/wiki/HTTP_compression#Content-Encoding_tokens

Is this understanding correct? Note that not only UTF-8, any values fails with same error.

So, asking the consumer not to send this header or removing this header before it reaches Apigee is the only solution? Had it executed PreFlow we could've done something in Apigee to remove it but here it looks a little weird that Apigee doesn't execute anything.

PFA debug trace where it fails with the said error, another interesting thing to note is that this header doesn't show up in Request Headers section. When it's not sent, flow works normally.

Thanks, appreciate your responses in this matter.

@kurtkanaskie @dchiesa1 @Sai Saran Vaidyanathan
@dknezic @ganadurai @Harish123 @Manisha_Chennu @Peeyush_Singhai @markjkelly 

[dchiesa1 · Accepted solution]

Yes, I think Apigee is behaving correctly in this case. I think your understanding is correct.

Also, in my opinion, yes, the easiest approach to avoid this problem is to persuade the developer of the client, to send a valid HTTP Message.  If for some reason that is not possible, then you would need to intercept the message before Apigee receives it - for example at an nginx gateway you control - and strip out the invalid header there. That seems like a poor idea though;  an expensive solution involving permanent infrastructure to a problem than can be solved by (probably) removal of a single line of errant code.  

[shrenikkumar-s]

Thanks @dchiesa1 for lightning fast response. Right now, we've put the poor idea as our solution for 2 reasons.

1. Consumer didn't want to make change based on the fact that it worked fine with Mulesoft 🙂

2. Since this can happen to any API proxy on our platform (by any of our consumers), we thought it's better to strip it off in WAF (F5/NGINX) rather than persuading various consumers.

Agree it's not a valid HTTP request, so Apigee deliberately creating an error? Was it same/similar in Apigee Edge?

[dchiesa1]

Yes, Apigee rejects broken requests. It was the same in Apigee Edge. Your search of the history here on the Community shows that this is not new. The q&A from 2016 predates the release of Apigee hybrid by 3+ years, and predate the release of Apigee X by 4+ years.

As for Mulesoft accepting requests with that invalid value in the Content-Encoding header, I have an opinion! Some situations are "grey", and there is some doubt because specifications are not always as precise as we would like.  For example, must the equals sign be percent-encoded in a URL path segment or not?  Recently I read the RFC 7230 and RFC 3986, and cannot determine whether equals SHOULD or SHOULD NOT be percent encoded in an HTTP request.  That's a grey area. I suppose a system taking either path, would be defensible.

The case with Content-Encoding is not grey. RFC 7231 is very clear: the spec defines these values: compress, deflate, and gzip.  Using or accepting anything other than those values seems like a bad idea; unnecessary, inappropriate, and likely to lead to confusion or sadness at some point. so if I were you I would file a bug on the Mulesoft product!  🙂

This is just, ...like, ....my opinion, man.