We use Apigee Cloud which sends the client request to our Cloud Run service which is a fastapi python server.
We are noticing that < 1% of our POST requests are ending up as Request Validation errors with body being empty. When clients retry the same request, it works.
Things I tried:
During the investigation, I added a new header in Apigee which would copy over the original content-length header into a new custom header and logged the request headers. I noticed that the new header had non-zero length indicating the size of the request. But the actual Content-Length header is 0 and body is empty
As a next step, I added a ExtractVariables policy in Apigee to check what is the actual request body, but I noticed the following:
Error parsing request payload. Cause: java.io.EOFException
I am unable to reproduce the issue outside reliably because it is happening for < 1% of requests and when the clients retry the request it succeeds. Can you please let me know how to debug this further?